Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36754

Use frag_percent, avg_item_size from indexer stats rather than ns_server calculating the values

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Currently, ns_server is using it's own calculation for computing "frag_percent" and "avg_item_size" values. Both these stats are dependent on data_size stat value. Any change to indexer data_size stat will impact the "frag_percent" and "avg_item_size" values. Also, "frag_percent" depends on "disk_size" value which has been changed as a part of MB-31787. This led to reporting incorrect fragmentation percentage on the UI (MB-36613)

      Instead of ns_server calculating these values, please use the values from indexer stats. Indexer periodic stats already exposes "frag_percent" stat. From build 6.5.0-4750, another stat "avg_item_size" is also exposed

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          I would like the indexing team to confirm that they are ok with the following behavior before we merge the patch.

          One of the reasons the calculation is performed by ns_server in the first place is so that these values are correct for partitioned indexes. With the proposed new approach, for partitioned indexes we will be forced to average the averages returned by the indexer. Which may be misleading at the very least.

          Aliaksey Artamonau Aliaksey Artamonau (Inactive) added a comment - I would like the indexing team to confirm that they are ok with the following behavior before we merge the patch. One of the reasons the calculation is performed by ns_server in the first place is so that these values are correct for partitioned indexes. With the proposed new approach, for partitioned indexes we will be forced to average the averages returned by the indexer. Which may be misleading at the very least.
          dfinlay Dave Finlay added a comment -

          Varun Velamuri: could you please respond to Aliaksey's question as to whether you are ok with the fact that the stats will show "average of the average" numbers? If so, please assign back to Steve W. and set the component to ns_server. If not we can leave things as they are.

          dfinlay Dave Finlay added a comment - Varun Velamuri : could you please respond to Aliaksey's question as to whether you are ok with the fact that the stats will show "average of the average" numbers? If so, please assign back to Steve W. and set the component to ns_server. If not we can leave things as they are.
          lynn.straus Lynn Straus added a comment -

          Added due date field (preset to Nov 15). Please update the due date to the current ETA for a fix. Thanks.

          lynn.straus Lynn Straus added a comment - Added due date field (preset to Nov 15). Please update the due date to the current ETA for a fix. Thanks.

          Thanks Aliaksey Artamonau , Dave Finlay  for pointing this out.

          The "data_size" indexer stat has been changed to report uncompressed data size as indexer doesn't have in-memory compression and the earlier number wasn't an accurate one.
          This makes it necessary to change the calculation of fragmentation and average item size stat in ns-server.

          The following can be done to avoid the problem with partitioned indexes:

          1. Indexer can report a new "data_size_on_disk" stat which can be used to calculate fragmentation.
          The current "data_size" in ns-server calculation can be replaced by "data_size_on_disk" to get the accurate number.

          2. For the avg_item_size, indexer already publishes a new stat "raw_data_size"(which doesn't include the mvcc versions) and can be used to calculate the accurate item size.
          For ns-server calculation, instead of data_size/items_count, it will be raw_data_size/items_count.

          I hope the new indexer stats for calculation do not need to be stored in ETS and shouldn't have an adverse impact on the memory consumption.

          Please let me know if these changes look feasible and we will add the "data_size_on_disk" stat.

          deepkaran.salooja Deepkaran Salooja added a comment - Thanks Aliaksey Artamonau  , Dave Finlay   for pointing this out. The "data_size" indexer stat has been changed to report uncompressed data size as indexer doesn't have in-memory compression and the earlier number wasn't an accurate one. This makes it necessary to change the calculation of fragmentation and average item size stat in ns-server. The following can be done to avoid the problem with partitioned indexes: 1. Indexer can report a new "data_size_on_disk" stat which can be used to calculate fragmentation. The current "data_size" in ns-server calculation can be replaced by "data_size_on_disk" to get the accurate number. 2. For the avg_item_size, indexer already publishes a new stat "raw_data_size"(which doesn't include the mvcc versions) and can be used to calculate the accurate item size. For ns-server calculation, instead of data_size/items_count, it will be raw_data_size/items_count. I hope the new indexer stats for calculation do not need to be stored in ETS and shouldn't have an adverse impact on the memory consumption. Please let me know if these changes look feasible and we will add the "data_size_on_disk" stat.

          Dave Finlay, Aliaksey Artamonau,

          Please let us know your thoughts on the approach suggested by Deepkaran Salooja.

          varun.velamuri Varun Velamuri added a comment - Dave Finlay , Aliaksey Artamonau , Please let us know your thoughts on the approach suggested by Deepkaran Salooja .

          Please let me know if these changes look feasible and we will add the "data_size_on_disk" stat.

          The approach is fine with me. The ns_server change will be more or less trivial. Please give the ticket back to us once the said stats are available.

          Aliaksey Artamonau Aliaksey Artamonau (Inactive) added a comment - Please let me know if these changes look feasible and we will add the "data_size_on_disk" stat. The approach is fine with me. The ns_server change will be more or less trivial. Please give the ticket back to us once the said stats are available.
          dfinlay Dave Finlay added a comment -

          For reasons of bug accounting removing ns_server for now; please reset the component area when you guys reassign.

          dfinlay Dave Finlay added a comment - For reasons of bug accounting removing ns_server for now; please reset the component area when you guys reassign.

          Aliaksey Artamonau, The new stats are now available in indexing master branch. 

          • For calculating avg_item_size, please use the stats raw_data_size and items_count. raw_data_size/items_count will give the average item size
          • For calculating frag_percent, please use the stats data_size_on_disk and log_space_on_diskdata_size_on_disk/log_space_on_disk should give the fragmentation percentage

          Please let us know if you require any further details from our side.

          varun.velamuri Varun Velamuri added a comment - Aliaksey Artamonau , The new stats are now available in indexing master branch.  For calculating avg_item_size , please use the stats raw_data_size and items_count . raw_data_size/items_count will give the average item size For calculating frag_percent , please use the stats data_size_on_disk and log_space_on_disk .  data_size_on_disk/log_space_on_disk should give the fragmentation percentage Please let us know if you require any further details from our side.

          Build couchbase-server-6.5.0-4823 contains indexing commit 2e4060b with commit message:
          MB-36754 Add data_size_on_disk, log_space_on_disk stats

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4823 contains indexing commit 2e4060b with commit message: MB-36754 Add data_size_on_disk, log_space_on_disk stats

          Build couchbase-server-7.0.0-1044 contains indexing commit 2e4060b with commit message:
          MB-36754 Add data_size_on_disk, log_space_on_disk stats

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1044 contains indexing commit 2e4060b with commit message: MB-36754 Add data_size_on_disk, log_space_on_disk stats

          The ns_server side of this change is: http://review.couchbase.org/#/c/117987/
          And I've attached a screenshot

          steve.watanabe Steve Watanabe added a comment - The ns_server side of this change is: http://review.couchbase.org/#/c/117987/ And I've attached a screenshot

          Varun Velamuri While testing this change a question came up. For this calculation:

          For calculating frag_percent, please use the stats data_size_on_disk and log_space_on_disk. data_size_on_disk/log_space_on_disk should give the fragmentation percentage
          

          I'm trying to understand the values used in the calculation...pardon if this elementary but I'm not familiar with index fragmentation. Would you please explain in layman's terms what are data_size_on_disk and log_space_on_disk. From the code review it says "DataSizeOnDisk is lss_data_size" but I'm not familiar with what lss_data_size is. What is log_space_on_disk?

          From the web I see there's two types of index fragmentation:
          1. Internal fragmentation occurs when data pages have excessive free space.
          2. External fragmentation occurs when data pages are out of order.

          which type of fragmentation is being calculated here?

          Also, the Index Stats displayed by the UI include:
          disk size: Total disk file size consumed by the index
          data size: Actual data size consumed by the index
          memory used: Total memory consumed by the index storage

          On my macbook, these have values of 11MB, 11.6MB, and 21MB, respectively. Does this mean the "disk size" is the compressed size of the "data size"? And what are the reasons the "memory used" are much larger than the "data size"?

          Thanks for the help in understanding this.

          steve.watanabe Steve Watanabe added a comment - Varun Velamuri While testing this change a question came up. For this calculation: For calculating frag_percent, please use the stats data_size_on_disk and log_space_on_disk. data_size_on_disk/log_space_on_disk should give the fragmentation percentage I'm trying to understand the values used in the calculation...pardon if this elementary but I'm not familiar with index fragmentation. Would you please explain in layman's terms what are data_size_on_disk and log_space_on_disk. From the code review it says "DataSizeOnDisk is lss_data_size" but I'm not familiar with what lss_data_size is. What is log_space_on_disk? From the web I see there's two types of index fragmentation: 1. Internal fragmentation occurs when data pages have excessive free space. 2. External fragmentation occurs when data pages are out of order. which type of fragmentation is being calculated here? Also, the Index Stats displayed by the UI include: disk size: Total disk file size consumed by the index data size: Actual data size consumed by the index memory used: Total memory consumed by the index storage On my macbook, these have values of 11MB, 11.6MB, and 21MB, respectively. Does this mean the "disk size" is the compressed size of the "data size"? And what are the reasons the "memory used" are much larger than the "data size"? Thanks for the help in understanding this.
          varun.velamuri Varun Velamuri added a comment - - edited

          Steve Watanabe, The frag_percent refers to external fragmentation.

          For each index, plasma (Plasma is the storage that GSI uses as a part of "Standard Global Secondary" storage mode) stores the compressed version of in-memory data on disk. Additionally, it stores some checkpoint files on disk. The size of the compressed data + size of checkpoint files constitutes the "disk_size" of an index.

          Plasma stores the data on disk in the form of "log structured storage". Due to this, if any of the pages are compacted, the compacted page will be added as a new page at the tail of the log and the old page will be discarded. lss_data_size refers to the size of valid data i.e. the size of data in pages that won't be discarded yet. 

          A component called lss_cleaner runs inside plasma storage to identify pages, compact them and mark them invalid. log_space_on_disk refers to all the pages on disk that are not marked for cleaning (or) pages on disk that will be considered for next iteration of cleaning. 

          varun.velamuri Varun Velamuri added a comment - - edited Steve Watanabe , The frag_percent refers to external fragmentation. For each index, plasma (Plasma is the storage that GSI uses as a part of "Standard Global Secondary" storage mode) stores the compressed version of in-memory data on disk. Additionally, it stores some checkpoint files on disk. The size of the compressed data + size of checkpoint files constitutes the "disk_size" of an index. Plasma stores the data on disk in the form of "log structured storage". Due to this, if any of the pages are compacted, the compacted page will be added as a new page at the tail of the log and the old page will be discarded. lss_data_size refers to the size of valid data i.e. the size of data in pages that won't be discarded yet.  A component called lss_cleaner runs inside plasma storage to identify pages, compact them and mark them invalid. log_space_on_disk refers to all the pages on disk that are not marked for cleaning (or) pages on disk that will be considered for next iteration of cleaning. 
          varun.velamuri Varun Velamuri added a comment - - edited

          In the example that you tried, I see that disk_size is 11MB, data_size is 11.6MB and memory_used by the index is 21MB.

          For each mutation that get's indexed, there is a plasma overhead for that mutation. E.g., before a page is compacted, it is around 52 bytes. After a page is compacted, the overhead is around 20 bytes. So, the memory_used upper limit should be around (items_count * (52-20) * 2 + data_size) (The 2 is required because for each index, we maintain 2 stores - Main store and back_store) of index. Similarly, the lower limit can be estimated to  data_size(data_size already accounts for plasma overhead after compaction). It is possible that some items in a page are compacted and some are not. So, it is difficult to validate the exact value of memory_used with data_size stat. However, it should be with-in the bounds.

          The 11MB of disk_size is the compacted and compressed version of 21MB of memory_used by the index.

          The above mentioned calculations are made assuming that index is completely resident in memory i.e. resident_percent is 100%. These calculations change if resident_percent is less than 100%

          varun.velamuri Varun Velamuri added a comment - - edited In the example that you tried, I see that disk_size is 11MB, data_size is 11.6MB and memory_used by the index is 21MB. For each mutation that get's indexed, there is a plasma overhead for that mutation. E.g., before a page is compacted, it is around 52 bytes. After a page is compacted, the overhead is around 20 bytes. So, the memory_used upper limit should be around (items_count * (52-20) * 2 + data_size)  (The 2 is required because for each index, we maintain 2 stores - Main store and back_store) of index. Similarly, the lower limit can be estimated to  data_size (data_size already accounts for plasma overhead after compaction). It is possible that some items in a page are compacted and some are not. So, it is difficult to validate the exact value of memory_used with data_size stat. However, it should be with-in the bounds. The 11MB of disk_size is the compacted and compressed version of 21MB of memory_used by the index. The above mentioned calculations are made assuming that index is completely resident in memory i.e. resident_percent is 100%. These calculations change if resident_percent is less than 100%

          Varun Velamuri, I think that the fragmention calculation you provided is incorrect. Can you please confirm?

          The way I understand the stats, data_size_on_disk is the size of valid data on disk, while log_space_on_disk is more or less the total size of disk storage being used (by valid data and garbage). So in the fully compacted case where there's no garbage on disk, you'd expect to see data_size_on_disk be equal to log_space_on_disk. You'd also expect to see no fragmentation (or 0%). Your calculation, though, will give a fragmentation of 100%.

          Aliaksey Artamonau Aliaksey Artamonau (Inactive) added a comment - Varun Velamuri , I think that the fragmention calculation you provided is incorrect. Can you please confirm? The way I understand the stats, data_size_on_disk is the size of valid data on disk, while log_space_on_disk is more or less the total size of disk storage being used (by valid data and garbage). So in the fully compacted case where there's no garbage on disk, you'd expect to see data_size_on_disk be equal to log_space_on_disk . You'd also expect to see no fragmentation (or 0%). Your calculation, though, will give a fragmentation of 100%.

          Build couchbase-server-6.5.0-4844 contains ns_server commit 479702c with commit message:
          MB-36754 Use indexer provided stats in calculations

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4844 contains ns_server commit 479702c with commit message: MB-36754 Use indexer provided stats in calculations

          Aliaksey Artamonau, You are right. I am wrong with my calculation. Fragmentation is ((log_space_on_disk - data_size_on_disk)/log_space_on_disk). Thanks for pointing this out.

          varun.velamuri Varun Velamuri added a comment - Aliaksey Artamonau , You are right. I am wrong with my calculation. Fragmentation is  ((log_space_on_disk - data_size_on_disk)/log_space_on_disk) . Thanks for pointing this out.

          Build couchbase-server-7.0.0-1050 contains ns_server commit 479702c with commit message:
          MB-36754 Use indexer provided stats in calculations

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1050 contains ns_server commit 479702c with commit message: MB-36754 Use indexer provided stats in calculations

          Verified on 6.5.0-4960

          mihir.kamdar Mihir Kamdar (Inactive) added a comment - Verified on 6.5.0-4960

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            varun.velamuri Varun Velamuri
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty