Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41924

[System Test] Very high fragmentation leading to disk full issues on KV nodes

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-3291
      Test : -test tests/2i/test_idx_clusterops_cheshire_cat.yml -scope tests/2i/scope_idx_cheshire_cat.yml
      Scale : 2
      Iteration : 1st

      KV bucket fragmentation is very high for all 3 buckets. There are 4 KV nodes in the cluster. Disk is full for all the 4 nodes due to this issue.

      CPU usage has also been consistently high (>80%) almost around the same time as fragmentation ratio getting to 90%.

      From cbstats from 172.23.104.16 for bucket1:
      ep_db_data_size: 2745713144
      ep_db_file_size: 37351228928
      curr_items: 1706190
      curr_items_tot: 3414827

      Attaching logs from around 12.30 AM 10/8/20.

      See attached screenshots.

      The KV nodes in the cluster are 172.23.104.16, 172.23.104.17, 172.23.104.18, 172.23.104.19

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-41857 CouchKVStoreFileCache not "releasing" files after compaction

          • Major - Major loss of function.
          • Closed

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty