Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-17135

[FTS] FTS index of 1M docs consumes 33G disk while ES index consumes 181MB

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.5.0
    • 4.5.0
    • fts
    • Untriaged
    • Unknown

    Description

      Build
      4.5.0-823

      Testcase
      ./testrunner -i perf.ini -t fts.stable_topology_fts.StableTopFTS.test_query_type,items=1000000,num_queries=1000,query_types=match,match_phrase,cluster=D:F,skip-cleanup=True,compare_es=True,get-cbcollect-info=True

      Simple default index containing 1M docs of size <1kb, with no replicas.

      Easily reproducible.

      The size of the ES index per documentation at https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html
      says is available under "store" param which shows 181MB -

      http://10.3.4.191:9200/_stats returns -

      indices: {
      default_es_index: {
      primaries: {},
      total: {
      docs: {
      count: 1000000,
      deleted: 0
      },
      store: {
      size_in_bytes: 181095262,
      throttle_time_in_millis: 545254
      },
      

      while FTS pindexes add upto 33G -

      [root@loquat02 home]# du -h
      33G	./@fts
      579M	./@fts/default_index_3f6c4df5f980a565_7d5abc86.pindex
      577M	./@fts/default_index_3f6c4df5f980a565_4a179970.pindex
      580M	./@fts/default_index_3f6c4df5f980a565_1c3dcdf0.pindex
      578M	./@fts/default_index_3f6c4df5f980a565_d44c5f96.pindex
      655M	./@fts/default_index_3f6c4df5f980a565_185bbfd.pindex
      735M	./@fts/default_index_3f6c4df5f980a565_fdf850de.pindex
      579M	./@fts/default_index_3f6c4df5f980a565_6cdf0367.pindex
      576M	./@fts/default_index_3f6c4df5f980a565_d871967a.pindex
      661M	./@fts/default_index_3f6c4df5f980a565_4923bdb6.pindex
      578M	./@fts/default_index_3f6c4df5f980a565_ecb637fa.pindex
      733M	./@fts/default_index_3f6c4df5f980a565_c9f4299b.pindex
      813M	./@fts/default_index_3f6c4df5f980a565_a7efcdb2.pindex
      578M	./@fts/default_index_3f6c4df5f980a565_7167756a.pindex
      661M	./@fts/default_index_3f6c4df5f980a565_abd2045e.pindex
      578M	./@fts/default_index_3f6c4df5f980a565_b52b2ee0.pindex
      579M	./@fts/default_index_3f6c4df5f980a565_1000041c.pindex
      737M	./@fts/default_index_3f6c4df5f980a565_67e7bd4a.pindex
      807M	./@fts/default_index_3f6c4df5f980a565_6a5dba01.pindex
      577M	./@fts/default_index_3f6c4df5f980a565_ce6bee05.pindex
      578M	./@fts/default_index_3f6c4df5f980a565_feb90c6d.pindex
      660M	./@fts/default_index_3f6c4df5f980a565_db87211.pindex
      581M	./@fts/default_index_3f6c4df5f980a565_77e5cc0c.pindex
      577M	./@fts/default_index_3f6c4df5f980a565_2c4e74e.pindex
      575M	./@fts/default_index_3f6c4df5f980a565_cf7690fd.pindex
      808M	./@fts/default_index_3f6c4df5f980a565_1f7c9143.pindex
      660M	./@fts/default_index_3f6c4df5f980a565_bf945e6a.pindex
      578M	./@fts/default_index_3f6c4df5f980a565_d2cee6f0.pindex
      659M	./@fts/default_index_3f6c4df5f980a565_7e1be035.pindex
      808M	./@fts/default_index_3f6c4df5f980a565_60e2ca8b.pindex
      152M	./@fts/default_index_3f6c4df5f980a565_1fbc862b.pindex
      579M	./@fts/default_index_3f6c4df5f980a565_b916e70c.pindex
      575M	./@fts/default_index_3f6c4df5f980a565_722629d9.pindex
      579M	./@fts/default_index_3f6c4df5f980a565_ba57bbbf.pindex
      804M	./@fts/default_index_3f6c4df5f980a565_a22c2867.pindex
      579M	./@fts/default_index_3f6c4df5f980a565_240c7d59.pindex
      579M	./@fts/default_index_3f6c4df5f980a565_90a2e844.pindex
      737M	./@fts/default_index_3f6c4df5f980a565_db30cac9.pindex
      733M	./@fts/default_index_3f6c4df5f980a565_607744fc.pindex
      736M	./@fts/default_index_3f6c4df5f980a565_1abf7496.pindex
      810M	./@fts/default_index_3f6c4df5f980a565_c5c9e077.pindex
      576M	./@fts/default_index_3f6c4df5f980a565_d70d0325.pindex
      580M	./@fts/default_index_3f6c4df5f980a565_6f9e5fd4.pindex
      578M	./@fts/default_index_3f6c4df5f980a565_a4ae9101.pindex
      736M	./@fts/default_index_3f6c4df5f980a565_ab9f8852.pindex
      736M	./@fts/default_index_3f6c4df5f980a565_4956c5c3.pindex
      657M	./@fts/default_index_3f6c4df5f980a565_fb7ae9b8.pindex
      579M	./@fts/default_index_3f6c4df5f980a565_a89358ed.pindex
      578M	./@fts/default_index_3f6c4df5f980a565_707029b.pindex
      576M	./@fts/default_index_3f6c4df5f980a565_96205122.pindex
      657M	./@fts/default_index_3f6c4df5f980a565_ef92ea2.pindex
      578M	./@fts/default_index_3f6c4df5f980a565_93e3b4f7.pindex
      579M	./@fts/default_index_3f6c4df5f980a565_cab57528.pindex
      

      Adding diag-info.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sandip.nandi Sandip Nandi (Inactive)
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty