Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55897

[CDC] KV is setting history_retention_bytes per kvstore to a small value (actual value as per calculation is 14.5 Gb/vb but KV is setting to 10Gb/vb), which in turn means maintaining lesser history

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • 7.2.0
    • couchbase-bucket
    • 7.2.0-5230

    Description

      Steps to Recreate:

      1. Create a 3 node cluster
      2. Create a magma buckets with (bucket_history_retention_seconds=259200,bucket_history_retention_bytes=10000000000000,) (vbuckets = 1024, replicas = 1)
      3. SInce 10TB(10000000000000) is the history per node and on niode 172.23.110.64 there are 683 vbuckets(including replica vbs), per vb history size should be 10000000000000/683 = 14641288433.4(approx 14.5 GB)
      4. but when checked in memcached logs we are setting it to 9765625000 (close to 10G)
      5. Observed history window starts getting moved when data is close to 18 to 20TB(because of 10Gb/vb,) although the expectation is to hold close to 30Tb data(15Gb/node) 

        -bash-4.2# /opt/couchbase/bin/cbstats -u Administrator -p password localhost:11210 -b GleamBookUsers0 all | grep reten
        		 
         ep_history_retention_bytes:                                10000000000000
        		 
         ep_history_retention_seconds:                              259200


        -bash-4.2#  /opt/couchbase/bin/cbstats -u Administrator -p password localhost:11210 -b GleamBookUsers0 vbucket | wc -l
        		 
        683

      Snippet from Memcached:

      memcached.log.000002.txt:2023-03-08T01:06:16.177650-08:00 INFO [(GleamBookUsers0) magma_65]Magma::SetHistoryRetentionSize changing history size window from 9765625000 bytes to 9765625000 bytes
      		 
      memcached.log.000002.txt:2023-03-08T01:06:16.177655-08:00 INFO [(GleamBookUsers0) magma_66]Magma::SetHistoryRetentionSize changing history size window from 9765625000 bytes to 9765625000 bytes
      		 
      memcached.log.000002.txt:2023-03-08T01:06:16.177657-08:00 INFO [(GleamBookUsers0) magma_67]Magma::SetHistoryRetentionSize changing history size window from 9765625000 bytes to 9765625000 bytes
      		 
      memcached.log.000002.txt:2023-03-08T01:06:16.177663-08:00 INFO [(GleamBookUsers0) magma_68]Magma::SetHistoryRetentionSize changing history size window from 9765625000 bytes to 9765625000 bytes
      		 
      memcached.log.000002.txt:2023-03-08T01:06:16.177666-08:00 INFO [(GleamBookUsers0) magma_69]Magma::SetHistoryRetentionSize changing history size window from 9765625000 bytes to 9765625000 bytes
      		 
      memcached.log.000002.txt:2023-03-08T01:06:16.177668-08:00 INFO [(GleamBookUsers0) magma_70]Magma::SetHistoryRetentionSize changing history size window from 9765625000 bytes to 9765625000 bytes
      		 
      memcached.log.000002.txt:2023-03-08T01:06:16.177671-08:00 INFO [(GleamBookUsers0) magma_71]Magma::SetHistoryRetentionSize changing history size window from 9765625000 bytes to 9765625000 byte

      Had a discussion with Paolo Cocchi on this, Based on his suggestion logging this ticket for further investigation

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ankush.sharma Ankush Sharma
            ankush.sharma Ankush Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty