Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55931

[Magma-CDC] Magma is holding longer duration history than the configured hisotry_retention_seconds (since last sstable has not reached 5% of the total data, so it keeps holding data)

    XMLWordPrintable

Details

    • 0

    Description

      Steps To Recreate:
      Steps to Recreate:

      1. Create a 3 node cluster
      2. Create a magma buckets with (bucket_history_retention_seconds=259200,bucket_history_retention_bytes=10000000000000,) (vbuckets = 1024, replicas = 1)
      3. Start data loading,
      4. Rebalance in one node
      5. When disk usage is around 32 Tb, start rebalancing out one node
      6. Pause rebalance and change bucket_history_retention_seconds=300
      7. Waited long enough (for more than 30 mins, infact it was same after 5 hours), and observed oldest document in magma history is 2 days old(calculated thru CAS)(though start seq number got moved as expected, becz of changes bucket_history_retention_seconds)(checked for vb_0)
      8. 32TB of data with one replica. (2048 vbcukets) so each vbucket size is 15GB
      9. sstable will be 5% of that so 750MB, sstable was not hitting 750MB, so holding on to longer duration history
      10. Had a discussion with Apaar, and showed the cluster to him, based on the discussion logging an improvement bug.

      Logs:
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1678605553/collectinfo-2023-03-12T071915-ns_1%40172.23.110.69.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1678605553/collectinfo-2023-03-12T071915-ns_1%40172.23.110.65.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1678605553/collectinfo-2023-03-12T071915-ns_1%40172.23.110.64.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1678605553/collectinfo-2023-03-12T071915-ns_1%40172.23.110.67.zip
      https://cb-engineering.s3.amazonaws.com/seq_history_300_Second_3.txt
      https://cb-engineering.s3.amazonaws.com/treedata_300.txt

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              apaar.gupta Apaar Gupta
              ankush.sharma Ankush Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty