Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45981

Index disk usage is bloating up massively for indexes moving during rebalance out indexer node. 6 times the actual data in bucket.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • Cheshire-Cat
    • storage-engine
    • 7.0.0-5032

    Description

      Steps:

      1. create a 1kv, 1 n1ql, 2 index nodes cluster
      2. start a expiry data load with ttl=1000
      3. ops is close to 50-80k
      4. rebalance in 1 index node. done is seconds as expected
      5. rebalance out 1 indexer node. indexes moving to other nodes is continuously bloating up in terms of disk space and due to that rebalance is not able to finish and continuously running.

      Bucket details:

      Moving indexes details:

      The indexes disk usage is ~200GB while bucket total disk usage is 30GB

      QE test

      git fetch "http://review.couchbase.org/TAF" refs/changes/34/151634/58 && git checkout FETCH_HEAD
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=couchstore,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.test_rebalance,nodes_init=1,graceful=True,skip_cleanup=True,num_items=500000000,num_buckets=1,bucket_names=GleamBook,doc_size=256,bucket_type=membase,eviction_policy=fullEviction,iterations=100,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=2,maxttl=1000,num_indexes=50,pc=1,sdk_client_pool=True,max_clients=10,index_nodes=2,query_nodes=1,cbas_nodes=0,fts_nodes=0 -m rest'
      

      Attachments

        1. Cliend_side_logs_TTL_disk_size_issue.txt
          3.81 MB
          Sanjit Chauhan
        2. default_0_15295270197879477213_main_lss_disk_size.png
          55 kB
          Saptarshi Sen
        3. default_default__default__default_0_15295270197879477213_main.png
          125 kB
          Saptarshi Sen
        4. default_Scope1_FunctionCollection0_0_2345227892394814798_main_lss_disk_size.png
          64 kB
          Saptarshi Sen
        5. default_Scope1_FunctionCollection0_default_Scope1_FunctionCollection0_0_2345227892394814798_main.png
          136 kB
          Saptarshi Sen
        6. Disk_size_going_down.png
          317 kB
          Sanjit Chauhan
        7. Disk_Size_not_going_down.png
          313 kB
          Sanjit Chauhan
        8. GleamBookUsers0__default_VolumeCollection0_idx3_18257027072023385584.main-1.png
          127 kB
          Saptarshi Sen
        9. GleamBookUsers0-_default-VolumeCollection0-idx39-12561421980180482790.main-1.png
          129 kB
          Saptarshi Sen
        10. image-2021-04-28-22-46-22-466.png
          95 kB
          Ritesh Agarwal
        11. image-2021-04-28-22-46-57-538.png
          466 kB
          Ritesh Agarwal
        12. Screenshot 2022-01-11 at 3.22.12 PM.png
          317 kB
          Sanjit Chauhan
        13. Screenshot 2022-01-11 at 3.22.12 PM-1.png
          317 kB
          Sanjit Chauhan
        14. Screenshot 2022-01-11 at 3.26.57 PM.png
          313 kB
          Sanjit Chauhan
        15. TTL_Zigzag_disk_size_graph.png
          503 kB
          Sanjit Chauhan
        16. with_compact_fullmarshal.png
          165 kB
          Saptarshi Sen
        17. without_compact_full_marshal.png
          114 kB
          Saptarshi Sen
        For Gerrit Dashboard: MB-45981
        # Subject Branch Project Status CR V

        Activity

          People

            sanjit.chauhan Sanjit Chauhan (Inactive)
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty