Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32655

improve projector debuggability

    XMLWordPrintable

Details

    • Untriaged
    • No

    Description

      As seen in recent customer case, some of the logging/stats improvements can help in quicker turnaround for projector issues:

      1. Increase logging frequency for KVDT, DCPT stats. Right now the logging is every 5min and the stream request timeout is also 5mins. So effectively there is only 1 logging statement before timeout. More frequent stats are helpful to deduce things and also correlate with memcached/indexer logs. 1min should be a better frequency.

      2. Log the mutations queued up in the the channels for different projector components. These can give a concrete picture of how badly the things are backed up and differentiate from the case where data is not coming from memcached.

      3. Endpoint latency stats (ENDP) do not seem to be adding up with the projector throughput. Need to verify if these stats are correct. Actually it would be good to verify all the endpoint stats.

      4. Projector currently store the time a mutation arrives as Ctime. We could propagate this time to Indexer and have stream reader calculate the latency it is taking for mutations to propagate from projector to indexer.

      5. Overall per topic(per bucket) stats equivalent to num_docs_pending, num_docs_queued and num_docs_processed. It would be good for projector to have unified stats logging similar to Indexer rather than component based logging. It makes it easy to make sense of overall picture and also allows for tools like mortimer to parse/plot the logs better.

      6. Add stats to calculate latency from projector/dcp to slice writer

      7. Integrate index evaluator stats into logging

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            girish.benakappa Girish Benakappa
            deepkaran.salooja Deepkaran Salooja
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty