Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34592

[CX] Investigate Periodic Dataset Flush Impact on Ingestion

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Major
    • 6.5.0
    • 6.5.0
    • analytics

    Description

      In Mad-Hatter, we started periodically flushing (checkpointing) datasets from memory to disk even when their memory budget is not full. This is done to ensure that datasets which have only few mutations will persist the latest DCP state. Currently the default setting is to flush datasets every 10 minutes. We need to investigate the impact of this periodic flush on ingestion.

      The period in which datasets are flushed can be configured using the Analytics service parameter (txnDatasetCheckpointInterval) by setting its value in (seconds). We need to run the ingestion performance experiments while setting txnDatasetCheckpointInterval to a large value (e.g. 86400) so that the periodic flush is never triggered.

      To do that, after the Analytics service is up, we need to do the following:

      1. Use the Analytics service configuration API to change the value of txnDatasetCheckpointInterval to 86400.
      2. Restart the Analytics service using the Analytics service restart API for the service configuration change to take effect.
      3. Run the ingestion experiments.

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              korrigan.clark Korrigan Clark (Inactive)
              murtadha.hubail Murtadha Hubail
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty