Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52981

Investigate: Why is there a data-size imbalance on cluster during and after load when compaction is disabled?

    XMLWordPrintable

Details

    • Task
    • Resolution: Unresolved
    • Major
    • Morpheus
    • 7.1.0
    • couchbase-bucket
    • None
    • 1

    Description

      I am using the python SDK 4.0.2 to bulk load data directly into KV using the upsert_multi() API. I am loading 5000 warehouses of ch2 (tpcc based) data into a five data node cluster. 5000w is approximately 500GB of data, with one replica it is about 1TB. What we are seeing is that one node has consistently higher disk usage than the other four nodes (see attached screenshot).

      Also, when I look at the UI metrics, the bucket shows about 3.05TB of disk utilization, but the scope shows 493GB (this seems correct for 0 replicas) of disk utilization. How are these numbers reported, and how do I interpret them? The bucket has only one scope.

      This is on Enterprise Edition 7.1.0 build 2556.
      Server node: 172.23.97.21
      Cluster logs are here:

      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2022-07-13T235935-ns_1%40172.23.97.21.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2022-07-13T235935-ns_1%40172.23.97.22.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2022-07-13T235935-ns_1%40172.23.97.23.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2022-07-13T235935-ns_1%40172.23.97.24.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2022-07-13T235935-ns_1%40172.23.97.25.zip

      Attachments

        1. checkpoint-memory-21.png
          checkpoint-memory-21.png
          102 kB
        2. checkpoint-memory-25.png
          checkpoint-memory-25.png
          98 kB
        3. disk_queue_items.png
          disk_queue_items.png
          237 kB
        4. KV-CPU.png
          KV-CPU.png
          118 kB
        5. screenshot-1.png
          screenshot-1.png
          41 kB
        6. Screen Shot 2022-07-14 at 9.32.03 AM.png
          Screen Shot 2022-07-14 at 9.32.03 AM.png
          2.28 MB
        7. Screen Shot 2022-07-15 at 10.32.57 AM.png
          Screen Shot 2022-07-15 at 10.32.57 AM.png
          172 kB
        8. Screen Shot 2022-07-15 at 10.36.03 AM.png
          Screen Shot 2022-07-15 at 10.36.03 AM.png
          140 kB
        9. Screenshot 2022-07-20 at 11.00.33.png
          Screenshot 2022-07-20 at 11.00.33.png
          1.17 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            bo-chun.wang Bo-Chun Wang
            vijay.sarathy Vijay Sarathy
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty