Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47674

[BP of MB-47666] Provide configurable option to disbale logging numDocsPending stat for projector

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Projector logs numDocsPending stats every 1min. To that extent, is spawns the go-routines to fetch seqnos from KV node. One go-routine queries ns_server every 10sec to validate the presence of a bucket in the cluster and clear the book-keeping if the bucket wa to be deleted.

      In cases where there are large number of buckets, periodically querying all the bucket endpoints is adding significant load on ns_server i.e. in a 30 bucket setup, it would be 30*6 = 180 requests per min. These requests would be to "pools/default/buckets/<bucket_name>?uuid=<>" endpoint which is a relatively costly call.

      The goal of this ticket is to provide a configurable option to disable the logging of num_docs_pending stat. Logging this stat is disabled by default and it can be enabled only if some real need for it arises

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-47674
          # Subject Branch Project Status CR V

          Activity

            varun.velamuri Varun Velamuri created issue -
            varun.velamuri Varun Velamuri made changes -
            Field Original Value New Value
            Link This issue Clones MB-47666 [ MB-47666 ]
            varun.velamuri Varun Velamuri made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            Fix Version/s Neo [ 17615 ]
            varun.velamuri Varun Velamuri made changes -
            Affects Version/s 7.0.1 [ 17104 ]
            Affects Version/s Neo [ 17615 ]

            Wayne Siu, Mihir Kamdar Requesting your approval for the inclusion of this ticket to 6.6.3. The change would be a very localised one and it would disable a stat logging by default at projector. Having this would help reduce periodic load on ns_server.

            CC: Jeelan Poola, Deepkaran Salooja, Ritam Sharma

            varun.velamuri Varun Velamuri added a comment - Wayne Siu , Mihir Kamdar Requesting your approval for the inclusion of this ticket to 6.6.3. The change would be a very localised one and it would disable a stat logging by default at projector. Having this would help reduce periodic load on ns_server. CC: Jeelan Poola , Deepkaran Salooja , Ritam Sharma

            Hi Varun Velamuri I am afraid its too late for 6.6.3. We have identified the potentially final RC for 6.6.3. If there is any other issue that warrants a new build for 6.6.3, we will consider this.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Hi Varun Velamuri I am afraid its too late for 6.6.3. We have identified the potentially final RC for 6.6.3. If there is any other issue that warrants a new build for 6.6.3, we will consider this.
            wayne Wayne Siu made changes -
            Fix Version/s 6.6.4 [ 17614 ]
            Fix Version/s 6.6.3 [ 17420 ]
            wayne Wayne Siu made changes -
            Link This issue is a backport of MB-47666 [ MB-47666 ]
            wayne Wayne Siu made changes -
            Link This issue Clones MB-47666 [ MB-47666 ]
            wayne Wayne Siu made changes -
            Assignee Varun Velamuri [ varun.velamuri ] Jeelan Poola [ jeelan.poola ]
            varun.velamuri Varun Velamuri made changes -
            Link This issue relates to CBSE-10484 [ CBSE-10484 ]
            varun.velamuri Varun Velamuri made changes -
            Link This issue relates to CBSE-10429 [ CBSE-10429 ]
            wayne Wayne Siu made changes -
            Labels 6.6.4-candidate
            jeelan.poola Jeelan Poola made changes -
            Assignee Jeelan Poola [ jeelan.poola ] Varun Velamuri [ varun.velamuri ]
            wayne Wayne Siu made changes -
            Labels 6.6.4-candidate 6.6.4-candidate approved-for-6.6.4
            wayne Wayne Siu made changes -
            Link This issue relates to MB-48997 [ MB-48997 ]
            wayne Wayne Siu made changes -
            Link This issue blocks MB-47673 [ MB-47673 ]

            Wayne Siu , yes. We will BP the change today.

            varun.velamuri Varun Velamuri added a comment - Wayne Siu , yes. We will BP the change today.
            varun.velamuri Varun Velamuri made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]

            Build couchbase-server-6.6.4-9948 contains indexing commit 4c2a7fa with commit message:
            MB-47674 : Increase polling interval in pollForDeletedBuckets

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.4-9948 contains indexing commit 4c2a7fa with commit message: MB-47674 : Increase polling interval in pollForDeletedBuckets
            wayne Wayne Siu made changes -
            Assignee Varun Velamuri [ varun.velamuri ] Ritam Sharma [ ritam.sharma ]
            wayne Wayne Siu made changes -
            Link This issue blocks MB-50048 [ MB-50048 ]
            ritam.sharma Ritam Sharma made changes -
            Assignee Ritam Sharma [ ritam.sharma ] Mihir Kamdar [ mihir.kamdar ]
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Assignee Mihir Kamdar [ mihir.kamdar ] Hemant Rajput [ hemant.rajput ]

            Validated on Enterprise Edition 6.6.5 build 10067

             

             

            [root@node1-cb660-centos7 logs]# grep pools/default/buckets/travel-sample?bucket http_access_internal.log | grep @projector-cbauth |tail 
            127.0.0.1 - @projector-cbauth [22/Dec/2021:06:31:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            127.0.0.1 - @projector-cbauth [22/Dec/2021:06:41:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            127.0.0.1 - @projector-cbauth [22/Dec/2021:06:51:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            127.0.0.1 - @projector-cbauth [22/Dec/2021:07:01:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            127.0.0.1 - @projector-cbauth [22/Dec/2021:07:11:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            127.0.0.1 - @projector-cbauth [22/Dec/2021:07:21:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            127.0.0.1 - @projector-cbauth [22/Dec/2021:07:31:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            127.0.0.1 - @projector-cbauth [22/Dec/2021:07:41:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1

            hemant.rajput Hemant Rajput added a comment - Validated on Enterprise Edition 6.6.5 build 10067     [root@node1-cb660-centos7 logs] # grep pools/default/buckets/travel-sample?bucket http_access_internal.log | grep @projector-cbauth |tail  127.0.0.1 - @projector-cbauth [22/Dec/2021:06:31:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:06:41:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:06:51:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:07:01:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:07:11:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:07:21:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:07:31:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:07:41:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            hemant.rajput Hemant Rajput made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            wayne Wayne Siu made changes -
            Link This issue relates to MB-48997 [ MB-48997 ]

            People

              hemant.rajput Hemant Rajput
              varun.velamuri Varun Velamuri
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty