Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44411

[BP MB-41788 to 6.6.2] - [System Test] : Negative value for items_count, backstore_raw_data_size index stat

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-3154
      Test : -test tests/2i/test_idx_steady_state_cheshire_cat.yml -scope tests/2i/scope_idx_cheshire_cat_moi.yml
      GSI Storage Mode : MOI
      Scale : 2
      Iteration : 1

      Indexer stats show a lot of indexes with negative value for backstore_raw_data_size stat right after they are created and built.

      Will try to see if this issue can be reproduced with a smaller setup and scale, and update the ticket accordingly.

      This excerpt is from 172.23.120.77. Other indexer nodes are :172.23.120.86, 172.23.121.77, 172.23.123.24, 172.23.123.25, 172.23.123.26

      2020-09-29T17:20:06.029-07:00 index_bucket1:scope_5:coll_-1:idx1 (replica 1) {"avg_array_length":2,"avg_drain_rate":11,"avg_item_size":286,"avg_mutation_rate":4,"avg_ts_interval":15355816,"avg_ts_items_count":6,"backstore_raw_data_size":-10,"build_progress":28,"data_size"
      :3973818,"docid_count":6287,"flush_queue_size":7,"index_state":2,"items_count":17946,"key_size_distribution":{"(65-256)":29880},"key_size_stats_since":1601425154639026256,"last_rollback_time":"0","memory_used":4237872,"num_docs_indexed":10429,"num_docs_pending":4465676,"n
      um_docs_processed":1814686,"num_flush_queued":10436,"num_items_flushed":29878,"num_open_snapshots":1,"num_snapshots":733,"progress_stat_time":"1601425204952697747","raw_data_size":5133384,"recs_in_mem":17851,"since_last_snapshot":8547143}
      2020-09-29T17:31:06.226-07:00 index_bucket3:scope_5:coll_-7:idx1 (replica 1) {"avg_array_length":2,"avg_drain_rate":10,"avg_item_size":184,"avg_mutation_rate":3,"avg_ts_interval":365840822,"avg_ts_items_count":29,"backstore_raw_data_size":-1372,"build_progress":2,"data_si
      ze":384364,"docid_count":602,"items_count":1735,"key_size_distribution":{"(65-256)":1857},"key_size_stats_since":1601425810536230065,"memory_used":409648,"num_docs_indexed":1134,"num_docs_pending":7181402,"num_docs_processed":148906,"num_flush_queued":1134,"num_items_flus
      hed":2609,"num_open_snapshots":1,"num_snapshots":107,"progress_stat_time":"1601425860013140733","raw_data_size":319348,"recs_in_mem":1735,"since_last_snapshot":579166879}
      2020-09-29T17:32:06.230-07:00 index_bucket3:scope_5:coll_-2:idx1 (replica 2) {"avg_array_length":2,"avg_drain_rate":8,"avg_item_size":241,"avg_mutation_rate":2,"avg_ts_interval":328917000,"avg_ts_items_count":39,"backstore_raw_data_size":-2153,"build_progress":3,"data_siz
      e":1036972,"docid_count":1657,"items_count":4687,"key_size_distribution":{"(65-256)":6607},"memory_used":1106566,"num_docs_indexed":2942,"num_docs_pending":7831837,"num_docs_processed":322442,"num_flush_queued":2942,"num_items_flushed":7775,"num_snapshots":126,"progress_s
      tat_time":"1601425917717067455","raw_data_size":1134212,"recs_in_mem":4687,"since_last_snapshot":28966526}
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-44411
          # Subject Branch Project Status CR V

          Activity

            Thanks Jeelan Poola. Akhil Mundroy Srinath Duvuru what is the ETA for merging the back port?

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Thanks Jeelan Poola . Akhil Mundroy Srinath Duvuru what is the ETA for merging the back port?

            Mihir Kamdar We discussed this in scrum today and prefer to fix this for 6.6.3 as there is more checking and testing we would like to do. Akhil Mundroy will open a back port for 6.6.3. FYI  Jeelan Poola John Liang

            srinath.duvuru Srinath Duvuru added a comment - Mihir Kamdar  We discussed this in scrum today and prefer to fix this for 6.6.3 as there is more checking and testing we would like to do. Akhil Mundroy  will open a back port for 6.6.3. FYI  Jeelan Poola John Liang

            Thanks Srinath Duvuru. I will change the fix version to 6.6.3 then. Wayne Siu FYI..

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Thanks Srinath Duvuru . I will change the fix version to 6.6.3 then. Wayne Siu FYI..

            Build couchbase-server-6.6.2-9581 contains plasma commit b150434 with commit message:
            MB-44411: Pass rollback filter into merge sibling lookup

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9581 contains plasma commit b150434 with commit message: MB-44411 : Pass rollback filter into merge sibling lookup

            Verified on 6.6.2-9581 for both plasma and MOI with the system test run for ~2 days each.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Verified on 6.6.2-9581 for both plasma and MOI with the system test run for ~2 days each.

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              jeelan.poola Jeelan Poola
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty