Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta-2
    • Fix Version/s: 2.0
    • Component/s: storage-engine, view-engine
    • Security Level: Public
    • Labels:
      None

      Description

      When splitting nodes (because they overflow), instead of having all new nodes with a full size and one with a smaller size, try to leave them all not more than half-full.
      Experiments (including evperf datasets) showed that this is benefical for incremental index updates (random inserts/updates), as it makes fragmentation grow more slowly and speedups a little inserts/updates and btree lookups.

      Example:

      Before

      1> btree_bench:test(2000000, 500000, 1000, "/home/fdmanana/tmp/foo.couch").
      Creating btree with initial number of items 2000000, batch size 5000, random batches
      Btree created in 276553.336 ms
      Btree stats after creation:
      btree_size 16684244
      file_size 1317839559
      fragmentation 98.73397001280942
      kv_chunk_threshold 7168
      kp_chunk_threshold 6144
      kv_count 2000000
      kp_nodes 628
      kv_nodes 32865
      max_depth 4
      min_depth 4
      avg_depth 4.0
      depth_90_percentile 4
      depth_95_percentile 4
      depth_99_percentile 4
      max_reduction_size 8
      min_reduction_size 8
      avg_reduction_size 8.0
      reduction_size_90_percentile 8
      reduction_size_95_percentile 8
      reduction_size_99_percentile 8
      max_elements_per_kp_node 107
      min_elements_per_kp_node 1
      avg_elements_per_kp_node 53.3312101910828
      elements_per_kp_node_90_percentile 96
      elements_per_kp_node_95_percentile 99
      elements_per_kp_node_99_percentile 106
      max_elements_per_kv_node 162
      min_elements_per_kv_node 2
      avg_elements_per_kv_node 60.85501293169025
      elements_per_kv_node_90_percentile 131
      elements_per_kv_node_95_percentile 145
      elements_per_kv_node_99_percentile 159
      max_kp_node_size 4602
      min_kp_node_size 44
      avg_kp_node_size 2294.2420382165606
      kp_node_size_90_percentile 4129
      kp_node_size_95_percentile 4258
      kp_node_size_99_percentile 4559
      max_compressed_kp_node_size 2038
      min_compressed_kp_node_size 43
      avg_compressed_kp_node_size 1034.7420382165606
      compressed_kp_node_size_90_percentile 1825
      compressed_kp_node_size_95_percentile 1882
      compressed_kp_node_size_99_percentile 2010
      max_kv_node_size 4375
      min_kv_node_size 55
      avg_kv_node_size 1644.0853491556368
      kv_node_size_90_percentile 3538
      kv_node_size_95_percentile 3916
      kv_node_size_99_percentile 4294
      max_compressed_kv_node_size 1239
      min_compressed_kv_node_size 37
      avg_compressed_kv_node_size 479.60921953445916
      compressed_kv_node_size_90_percentile 991
      compressed_kv_node_size_95_percentile 1094
      compressed_kv_node_size_99_percentile 1193
      max_key_size 16
      min_key_size 16
      avg_key_size 16.0
      key_size_90_percentile 16
      key_size_95_percentile 16
      key_size_99_percentile 16
      max_value_size 6
      min_value_size 6
      avg_value_size 6.0
      value_size_90_percentile 6
      value_size_95_percentile 6
      value_size_99_percentile 6

      Doing key lookups for every 1000th key (2000 keys)
      Percentiles for single key lookups, 90% -> 1.539ms, 95% -> 6.763ms, 99% -> 12.805ms

      Compacting btree
      Btree compaction took 57355.094 ms
      Btree stats after compaction:
      btree_size 15269217
      file_size 15269217
      fragmentation 0.0
      kv_chunk_threshold 7168
      kp_chunk_threshold 6144
      kv_count 2000000
      kp_nodes 123
      kv_nodes 12500
      max_depth 4
      min_depth 4
      avg_depth 4.0
      depth_90_percentile 4
      depth_95_percentile 4
      depth_99_percentile 4
      max_reduction_size 8
      min_reduction_size 8
      avg_reduction_size 8.0
      reduction_size_90_percentile 8
      reduction_size_95_percentile 8
      reduction_size_99_percentile 8
      max_elements_per_kp_node 105
      min_elements_per_kp_node 2
      avg_elements_per_kp_node 102.6178861788618
      elements_per_kp_node_90_percentile 105
      elements_per_kp_node_95_percentile 105
      elements_per_kp_node_99_percentile 105
      max_elements_per_kv_node 160
      min_elements_per_kv_node 160
      avg_elements_per_kv_node 160.0
      elements_per_kv_node_90_percentile 160
      elements_per_kv_node_95_percentile 160
      elements_per_kv_node_99_percentile 160
      max_kp_node_size 4516
      min_kp_node_size 87
      avg_kp_node_size 4413.569105691057
      kp_node_size_90_percentile 4516
      kp_node_size_95_percentile 4516
      kp_node_size_99_percentile 4516
      max_compressed_kp_node_size 1686
      min_compressed_kp_node_size 72
      avg_compressed_kp_node_size 1347.2113821138212
      compressed_kp_node_size_90_percentile 1394
      compressed_kp_node_size_95_percentile 1401
      compressed_kp_node_size_99_percentile 1417
      max_kv_node_size 4321
      min_kv_node_size 4321
      avg_kv_node_size 4321.0
      kv_node_size_90_percentile 4321
      kv_node_size_95_percentile 4321
      kv_node_size_99_percentile 4321
      max_compressed_kv_node_size 1228
      min_compressed_kv_node_size 1183
      avg_compressed_kv_node_size 1199.90384
      compressed_kv_node_size_90_percentile 1210
      compressed_kv_node_size_95_percentile 1211
      compressed_kv_node_size_99_percentile 1214
      max_key_size 16
      min_key_size 16
      avg_key_size 16.0
      key_size_90_percentile 16
      key_size_95_percentile 16
      key_size_99_percentile 16
      max_value_size 6
      min_value_size 6
      avg_value_size 6.0
      value_size_90_percentile 6
      value_size_95_percentile 6
      value_size_99_percentile 6

      Doing key lookups for every 1000th key (2000 keys)
      Percentiles for single key lookups, 90% -> 1.395ms, 95% -> 1.567ms, 99% -> 1.88ms

      Starting incremental inserts of 500000 new items, in random batches of size 5000
      Incremental inserts took 36789.824 ms
      Btree stats after inserts:
      btree_size 19447730
      file_size 180187724
      fragmentation 89.20696173508468
      kv_chunk_threshold 7168
      kp_chunk_threshold 6144
      kv_count 2500000
      kp_nodes 320
      kv_nodes 20791
      max_depth 4
      min_depth 4
      avg_depth 4.0
      depth_90_percentile 4
      depth_95_percentile 4
      depth_99_percentile 4
      max_reduction_size 8
      min_reduction_size 8
      avg_reduction_size 8.0
      reduction_size_90_percentile 8
      reduction_size_95_percentile 8
      reduction_size_99_percentile 8
      max_elements_per_kp_node 106
      min_elements_per_kp_node 1
      avg_elements_per_kp_node 65.96875
      elements_per_kp_node_90_percentile 105
      elements_per_kp_node_95_percentile 105
      elements_per_kp_node_99_percentile 105
      max_elements_per_kv_node 162
      min_elements_per_kv_node 2
      avg_elements_per_kv_node 120.24433649175124
      elements_per_kv_node_90_percentile 160
      elements_per_kv_node_95_percentile 160
      elements_per_kv_node_99_percentile 160
      max_kp_node_size 4559
      min_kp_node_size 44
      avg_kp_node_size 2837.65625
      kp_node_size_90_percentile 4516
      kp_node_size_95_percentile 4516
      kp_node_size_99_percentile 4516
      max_compressed_kp_node_size 1995
      min_compressed_kp_node_size 44
      avg_compressed_kp_node_size 1029.378125
      compressed_kp_node_size_90_percentile 1394
      compressed_kp_node_size_95_percentile 1457
      compressed_kp_node_size_99_percentile 1551
      max_kv_node_size 4375
      min_kv_node_size 55
      avg_kv_node_size 3247.5970852772834
      kv_node_size_90_percentile 4321
      kv_node_size_95_percentile 4321
      kv_node_size_99_percentile 4321
      max_compressed_kv_node_size 1228
      min_compressed_kv_node_size 39
      avg_compressed_kv_node_size 911.1964792458275
      compressed_kv_node_size_90_percentile 1209
      compressed_kv_node_size_95_percentile 1210
      compressed_kv_node_size_99_percentile 1213
      max_key_size 16
      min_key_size 16
      avg_key_size 16.0
      key_size_90_percentile 16
      key_size_95_percentile 16
      key_size_99_percentile 16
      max_value_size 6
      min_value_size 6
      avg_value_size 6.0
      value_size_90_percentile 6
      value_size_95_percentile 6
      value_size_99_percentile 6

      Doing key lookups for every 1000th key (2500 keys)
      Percentiles for single key lookups, 90% -> 1.47ms, 95% -> 1.603ms, 99% -> 4.404ms

      Starting incremental updates of every 5th item in the btree, in batches of size 5000 (500000 items), random order
      Incremental updates took 61793.899 ms
      Btree stats after updates:
      btree_size 19447730
      file_size 180187724
      fragmentation 89.20696173508468
      kv_chunk_threshold 7168
      kp_chunk_threshold 6144
      kv_count 2500000
      kp_nodes 320
      kv_nodes 20791
      max_depth 4
      min_depth 4
      avg_depth 4.0
      depth_90_percentile 4
      depth_95_percentile 4
      depth_99_percentile 4
      max_reduction_size 8
      min_reduction_size 8
      avg_reduction_size 8.0
      reduction_size_90_percentile 8
      reduction_size_95_percentile 8
      reduction_size_99_percentile 8
      max_elements_per_kp_node 106
      min_elements_per_kp_node 1
      avg_elements_per_kp_node 65.96875
      elements_per_kp_node_90_percentile 105
      elements_per_kp_node_95_percentile 105
      elements_per_kp_node_99_percentile 105
      max_elements_per_kv_node 162
      min_elements_per_kv_node 2
      avg_elements_per_kv_node 120.24433649175124
      elements_per_kv_node_90_percentile 160
      elements_per_kv_node_95_percentile 160
      elements_per_kv_node_99_percentile 160
      max_kp_node_size 4559
      min_kp_node_size 44
      avg_kp_node_size 2837.65625
      kp_node_size_90_percentile 4516
      kp_node_size_95_percentile 4516
      kp_node_size_99_percentile 4516
      max_compressed_kp_node_size 1995
      min_compressed_kp_node_size 44
      avg_compressed_kp_node_size 1029.378125
      compressed_kp_node_size_90_percentile 1394
      compressed_kp_node_size_95_percentile 1457
      compressed_kp_node_size_99_percentile 1551
      max_kv_node_size 4375
      min_kv_node_size 55
      avg_kv_node_size 3247.5970852772834
      kv_node_size_90_percentile 4321
      kv_node_size_95_percentile 4321
      kv_node_size_99_percentile 4321
      max_compressed_kv_node_size 1228
      min_compressed_kv_node_size 39
      avg_compressed_kv_node_size 911.1964792458275
      compressed_kv_node_size_90_percentile 1209
      compressed_kv_node_size_95_percentile 1210
      compressed_kv_node_size_99_percentile 1213
      max_key_size 16
      min_key_size 16
      avg_key_size 16.0
      key_size_90_percentile 16
      key_size_95_percentile 16
      key_size_99_percentile 16
      max_value_size 6
      min_value_size 6
      avg_value_size 6.0
      value_size_90_percentile 6
      value_size_95_percentile 6
      value_size_99_percentile 6

      Doing key lookups for every 1000th key (2500 keys)
      Percentiles for single key lookups, 90% -> 1.053ms, 95% -> 1.232ms, 99% -> 4.041ms

      Compacting btree
      Btree compaction took 12494.651 ms
      Btree stats after compaction:
      btree_size 19088176
      file_size 19088176
      fragmentation 0.0
      kv_chunk_threshold 7168
      kp_chunk_threshold 6144
      kv_count 2500000
      kp_nodes 152
      kv_nodes 15625
      max_depth 4
      min_depth 4
      avg_depth 4.0
      depth_90_percentile 4
      depth_95_percentile 4
      depth_99_percentile 4
      max_reduction_size 8
      min_reduction_size 8
      avg_reduction_size 8.0
      reduction_size_90_percentile 8
      reduction_size_95_percentile 8
      reduction_size_99_percentile 8
      max_elements_per_kp_node 105
      min_elements_per_kp_node 2
      avg_elements_per_kp_node 103.78947368421052
      elements_per_kp_node_90_percentile 105
      elements_per_kp_node_95_percentile 105
      elements_per_kp_node_99_percentile 105
      max_elements_per_kv_node 160
      min_elements_per_kv_node 160
      avg_elements_per_kv_node 160.0
      elements_per_kv_node_90_percentile 160
      elements_per_kv_node_95_percentile 160
      elements_per_kv_node_99_percentile 160
      max_kp_node_size 4516
      min_kp_node_size 87
      avg_kp_node_size 4463.9473684210525
      kp_node_size_90_percentile 4516
      kp_node_size_95_percentile 4516
      kp_node_size_99_percentile 4516
      max_compressed_kp_node_size 1686
      min_compressed_kp_node_size 75
      avg_compressed_kp_node_size 1361.5394736842106
      compressed_kp_node_size_90_percentile 1392
      compressed_kp_node_size_95_percentile 1400
      compressed_kp_node_size_99_percentile 1417
      max_kv_node_size 4321
      min_kv_node_size 4321
      avg_kv_node_size 4321.0
      kv_node_size_90_percentile 4321
      kv_node_size_95_percentile 4321
      kv_node_size_99_percentile 4321
      max_compressed_kv_node_size 1228
      min_compressed_kv_node_size 1183
      avg_compressed_kv_node_size 1200.02208
      compressed_kv_node_size_90_percentile 1210
      compressed_kv_node_size_95_percentile 1211
      compressed_kv_node_size_99_percentile 1214
      max_key_size 16
      min_key_size 16
      avg_key_size 16.0
      key_size_90_percentile 16
      key_size_95_percentile 16
      key_size_99_percentile 16
      max_value_size 6
      min_value_size 6
      avg_value_size 6.0
      value_size_90_percentile 6
      value_size_95_percentile 6
      value_size_99_percentile 6

      Doing key lookups for every 1000th key (2500 keys)
      Percentiles for single key lookups, 90% -> 1.018ms, 95% -> 1.169ms, 99% -> 1.498ms

      ok
      2>

      After

      1> btree_bench:test(2000000, 500000, 1000, "/home/fdmanana/tmp/foo.couch").
      Creating btree with initial number of items 2000000, batch size 5000, random batches
      Btree created in 268182.733 ms
      Btree stats after creation:
      btree_size 16422278
      file_size 1218231774
      fragmentation 98.65195783343606
      kv_chunk_threshold 7168
      kp_chunk_threshold 6144
      kv_count 2000000
      kp_nodes 662
      kv_nodes 28530
      max_depth 4
      min_depth 4
      avg_depth 4.0
      depth_90_percentile 4
      depth_95_percentile 4
      depth_99_percentile 4
      max_reduction_size 8
      min_reduction_size 8
      avg_reduction_size 8.0
      reduction_size_90_percentile 8
      reduction_size_95_percentile 8
      reduction_size_99_percentile 8
      max_elements_per_kp_node 107
      min_elements_per_kp_node 1
      avg_elements_per_kp_node 44.095166163142
      elements_per_kp_node_90_percentile 78
      elements_per_kp_node_95_percentile 93
      elements_per_kp_node_99_percentile 102
      max_elements_per_kv_node 162
      min_elements_per_kv_node 1
      avg_elements_per_kv_node 70.10164738871363
      elements_per_kv_node_90_percentile 136
      elements_per_kv_node_95_percentile 146
      elements_per_kv_node_99_percentile 158
      max_kp_node_size 4602
      min_kp_node_size 44
      avg_kp_node_size 1897.0921450151056
      kp_node_size_90_percentile 3355
      kp_node_size_95_percentile 4000
      kp_node_size_99_percentile 4387
      max_compressed_kp_node_size 2225
      min_compressed_kp_node_size 43
      avg_compressed_kp_node_size 882.0135951661631
      compressed_kp_node_size_90_percentile 1529
      compressed_kp_node_size_95_percentile 1776
      compressed_kp_node_size_99_percentile 1977
      max_kv_node_size 4375
      min_kv_node_size 28
      avg_kv_node_size 1893.7444794952683
      kv_node_size_90_percentile 3673
      kv_node_size_95_percentile 3943
      kv_node_size_99_percentile 4267
      max_compressed_kv_node_size 1234
      min_compressed_kv_node_size 25
      avg_compressed_kv_node_size 546.8209954433929
      compressed_kv_node_size_90_percentile 1028
      compressed_kv_node_size_95_percentile 1103
      compressed_kv_node_size_99_percentile 1189
      max_key_size 16
      min_key_size 16
      avg_key_size 16.0
      key_size_90_percentile 16
      key_size_95_percentile 16
      key_size_99_percentile 16
      max_value_size 6
      min_value_size 6
      avg_value_size 6.0
      value_size_90_percentile 6
      value_size_95_percentile 6
      value_size_99_percentile 6

      Doing key lookups for every 1000th key (2000 keys)
      Percentiles for single key lookups, 90% -> 1.301ms, 95% -> 4.603ms, 99% -> 12.706ms

      Compacting btree
      Btree compaction took 62111.127 ms
      Btree stats after compaction:
      btree_size 15269217
      file_size 15269217
      fragmentation 0.0
      kv_chunk_threshold 7168
      kp_chunk_threshold 6144
      kv_count 2000000
      kp_nodes 123
      kv_nodes 12500
      max_depth 4
      min_depth 4
      avg_depth 4.0
      depth_90_percentile 4
      depth_95_percentile 4
      depth_99_percentile 4
      max_reduction_size 8
      min_reduction_size 8
      avg_reduction_size 8.0
      reduction_size_90_percentile 8
      reduction_size_95_percentile 8
      reduction_size_99_percentile 8
      max_elements_per_kp_node 105
      min_elements_per_kp_node 2
      avg_elements_per_kp_node 102.6178861788618
      elements_per_kp_node_90_percentile 105
      elements_per_kp_node_95_percentile 105
      elements_per_kp_node_99_percentile 105
      max_elements_per_kv_node 160
      min_elements_per_kv_node 160
      avg_elements_per_kv_node 160.0
      elements_per_kv_node_90_percentile 160
      elements_per_kv_node_95_percentile 160
      elements_per_kv_node_99_percentile 160
      max_kp_node_size 4516
      min_kp_node_size 87
      avg_kp_node_size 4413.569105691057
      kp_node_size_90_percentile 4516
      kp_node_size_95_percentile 4516
      kp_node_size_99_percentile 4516
      max_compressed_kp_node_size 1686
      min_compressed_kp_node_size 72
      avg_compressed_kp_node_size 1347.2113821138212
      compressed_kp_node_size_90_percentile 1394
      compressed_kp_node_size_95_percentile 1401
      compressed_kp_node_size_99_percentile 1417
      max_kv_node_size 4321
      min_kv_node_size 4321
      avg_kv_node_size 4321.0
      kv_node_size_90_percentile 4321
      kv_node_size_95_percentile 4321
      kv_node_size_99_percentile 4321
      max_compressed_kv_node_size 1228
      min_compressed_kv_node_size 1183
      avg_compressed_kv_node_size 1199.90384
      compressed_kv_node_size_90_percentile 1210
      compressed_kv_node_size_95_percentile 1211
      compressed_kv_node_size_99_percentile 1214
      max_key_size 16
      min_key_size 16
      avg_key_size 16.0
      key_size_90_percentile 16
      key_size_95_percentile 16
      key_size_99_percentile 16
      max_value_size 6
      min_value_size 6
      avg_value_size 6.0
      value_size_90_percentile 6
      value_size_95_percentile 6
      value_size_99_percentile 6

      Doing key lookups for every 1000th key (2000 keys)
      Percentiles for single key lookups, 90% -> 1.324ms, 95% -> 1.45ms, 99% -> 1.849ms

      Starting incremental inserts of 500000 new items, in random batches of size 5000
      Incremental inserts took 36604.463 ms
      Btree stats after inserts:
      btree_size 19431715
      file_size 171302728
      fragmentation 88.65650580882752
      kv_chunk_threshold 7168
      kp_chunk_threshold 6144
      kv_count 2500000
      kp_nodes 279
      kv_nodes 20419
      max_depth 4
      min_depth 4
      avg_depth 4.0
      depth_90_percentile 4
      depth_95_percentile 4
      depth_99_percentile 4
      max_reduction_size 8
      min_reduction_size 8
      avg_reduction_size 8.0
      reduction_size_90_percentile 8
      reduction_size_95_percentile 8
      reduction_size_99_percentile 8
      max_elements_per_kp_node 106
      min_elements_per_kp_node 1
      avg_elements_per_kp_node 74.18279569892474
      elements_per_kp_node_90_percentile 105
      elements_per_kp_node_95_percentile 105
      elements_per_kp_node_99_percentile 105
      max_elements_per_kv_node 162
      min_elements_per_kv_node 1
      avg_elements_per_kv_node 122.43498702189137
      elements_per_kv_node_90_percentile 160
      elements_per_kv_node_95_percentile 160
      elements_per_kv_node_99_percentile 160
      max_kp_node_size 4559
      min_kp_node_size 44
      avg_kp_node_size 3190.8602150537636
      kp_node_size_90_percentile 4516
      kp_node_size_95_percentile 4516
      kp_node_size_99_percentile 4516
      max_compressed_kp_node_size 2018
      min_compressed_kp_node_size 45
      avg_compressed_kp_node_size 1159.6810035842293
      compressed_kp_node_size_90_percentile 1592
      compressed_kp_node_size_95_percentile 1790
      compressed_kp_node_size_99_percentile 1923
      max_kv_node_size 4375
      min_kv_node_size 28
      avg_kv_node_size 3306.744649591067
      kv_node_size_90_percentile 4321
      kv_node_size_95_percentile 4321
      kv_node_size_99_percentile 4321
      max_compressed_kv_node_size 1230
      min_compressed_kv_node_size 29
      avg_compressed_kv_node_size 927.4621675890103
      compressed_kv_node_size_90_percentile 1209
      compressed_kv_node_size_95_percentile 1210
      compressed_kv_node_size_99_percentile 1213
      max_key_size 16
      min_key_size 16
      avg_key_size 16.0
      key_size_90_percentile 16
      key_size_95_percentile 16
      key_size_99_percentile 16
      max_value_size 6
      min_value_size 6
      avg_value_size 6.0
      value_size_90_percentile 6
      value_size_95_percentile 6
      value_size_99_percentile 6

      Doing key lookups for every 1000th key (2500 keys)
      Percentiles for single key lookups, 90% -> 1.115ms, 95% -> 1.264ms, 99% -> 3.103ms

      Starting incremental updates of every 5th item in the btree, in batches of size 5000 (500000 items), random order
      Incremental updates took 58408.701 ms
      Btree stats after updates:
      btree_size 19431715
      file_size 171302728
      fragmentation 88.65650580882752
      kv_chunk_threshold 7168
      kp_chunk_threshold 6144
      kv_count 2500000
      kp_nodes 279
      kv_nodes 20419
      max_depth 4
      min_depth 4
      avg_depth 4.0
      depth_90_percentile 4
      depth_95_percentile 4
      depth_99_percentile 4
      max_reduction_size 8
      min_reduction_size 8
      avg_reduction_size 8.0
      reduction_size_90_percentile 8
      reduction_size_95_percentile 8
      reduction_size_99_percentile 8
      max_elements_per_kp_node 106
      min_elements_per_kp_node 1
      avg_elements_per_kp_node 74.18279569892474
      elements_per_kp_node_90_percentile 105
      elements_per_kp_node_95_percentile 105
      elements_per_kp_node_99_percentile 105
      max_elements_per_kv_node 162
      min_elements_per_kv_node 1
      avg_elements_per_kv_node 122.43498702189137
      elements_per_kv_node_90_percentile 160
      elements_per_kv_node_95_percentile 160
      elements_per_kv_node_99_percentile 160
      max_kp_node_size 4559
      min_kp_node_size 44
      avg_kp_node_size 3190.8602150537636
      kp_node_size_90_percentile 4516
      kp_node_size_95_percentile 4516
      kp_node_size_99_percentile 4516
      max_compressed_kp_node_size 2018
      min_compressed_kp_node_size 45
      avg_compressed_kp_node_size 1159.6810035842293
      compressed_kp_node_size_90_percentile 1592
      compressed_kp_node_size_95_percentile 1790
      compressed_kp_node_size_99_percentile 1923
      max_kv_node_size 4375
      min_kv_node_size 28
      avg_kv_node_size 3306.744649591067
      kv_node_size_90_percentile 4321
      kv_node_size_95_percentile 4321
      kv_node_size_99_percentile 4321
      max_compressed_kv_node_size 1230
      min_compressed_kv_node_size 29
      avg_compressed_kv_node_size 927.4621675890103
      compressed_kv_node_size_90_percentile 1209
      compressed_kv_node_size_95_percentile 1210
      compressed_kv_node_size_99_percentile 1213
      max_key_size 16
      min_key_size 16
      avg_key_size 16.0
      key_size_90_percentile 16
      key_size_95_percentile 16
      key_size_99_percentile 16
      max_value_size 6
      min_value_size 6
      avg_value_size 6.0
      value_size_90_percentile 6
      value_size_95_percentile 6
      value_size_99_percentile 6

      Doing key lookups for every 1000th key (2500 keys)
      Percentiles for single key lookups, 90% -> 1.433ms, 95% -> 1.599ms, 99% -> 3.0ms

      Compacting btree
      Btree compaction took 12399.026 ms
      Btree stats after compaction:
      btree_size 19088176
      file_size 19088176
      fragmentation 0.0
      kv_chunk_threshold 7168
      kp_chunk_threshold 6144
      kv_count 2500000
      kp_nodes 152
      kv_nodes 15625
      max_depth 4
      min_depth 4
      avg_depth 4.0
      depth_90_percentile 4
      depth_95_percentile 4
      depth_99_percentile 4
      max_reduction_size 8
      min_reduction_size 8
      avg_reduction_size 8.0
      reduction_size_90_percentile 8
      reduction_size_95_percentile 8
      reduction_size_99_percentile 8
      max_elements_per_kp_node 105
      min_elements_per_kp_node 2
      avg_elements_per_kp_node 103.78947368421052
      elements_per_kp_node_90_percentile 105
      elements_per_kp_node_95_percentile 105
      elements_per_kp_node_99_percentile 105
      max_elements_per_kv_node 160
      min_elements_per_kv_node 160
      avg_elements_per_kv_node 160.0
      elements_per_kv_node_90_percentile 160
      elements_per_kv_node_95_percentile 160
      elements_per_kv_node_99_percentile 160
      max_kp_node_size 4516
      min_kp_node_size 87
      avg_kp_node_size 4463.9473684210525
      kp_node_size_90_percentile 4516
      kp_node_size_95_percentile 4516
      kp_node_size_99_percentile 4516
      max_compressed_kp_node_size 1686
      min_compressed_kp_node_size 75
      avg_compressed_kp_node_size 1361.5394736842106
      compressed_kp_node_size_90_percentile 1392
      compressed_kp_node_size_95_percentile 1400
      compressed_kp_node_size_99_percentile 1417
      max_kv_node_size 4321
      min_kv_node_size 4321
      avg_kv_node_size 4321.0
      kv_node_size_90_percentile 4321
      kv_node_size_95_percentile 4321
      kv_node_size_99_percentile 4321
      max_compressed_kv_node_size 1228
      min_compressed_kv_node_size 1183
      avg_compressed_kv_node_size 1200.02208
      compressed_kv_node_size_90_percentile 1210
      compressed_kv_node_size_95_percentile 1211
      compressed_kv_node_size_99_percentile 1214
      max_key_size 16
      min_key_size 16
      avg_key_size 16.0
      key_size_90_percentile 16
      key_size_95_percentile 16
      key_size_99_percentile 16
      max_value_size 6
      min_value_size 6
      avg_value_size 6.0
      value_size_90_percentile 6
      value_size_95_percentile 6
      value_size_99_percentile 6

      Doing key lookups for every 1000th key (2500 keys)
      Percentiles for single key lookups, 90% -> 0.943ms, 95% -> 1.106ms, 99% -> 1.396ms

      ok

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        For an evperf dataset, incrementally updating an index (ddoc with 4 views) that had 1 million items (per view) and a data size of 167Mb, with 1 million more new items (like it can happens after a rebalance-out):

          • Before this change

        Update duration: 26m50.270s
        Final file size: 12 513 Mb

          • After this change

        Update duration: 25m43.893s
        Final file size: 11 640 Mb

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - For an evperf dataset, incrementally updating an index (ddoc with 4 views) that had 1 million items (per view) and a data size of 167Mb, with 1 million more new items (like it can happens after a rebalance-out): Before this change Update duration: 26m50.270s Final file size: 12 513 Mb After this change Update duration: 25m43.893s Final file size: 11 640 Mb
        Hide
        steve Steve Yen added a comment -

        moved to 2.0.2 per bug-scrub mtg

        Show
        steve Steve Yen added a comment - moved to 2.0.2 per bug-scrub mtg
        Hide
        steve Steve Yen added a comment -

        moved back to 2.0 per bug-scrub mtg

        Show
        steve Steve Yen added a comment - moved back to 2.0 per bug-scrub mtg
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #537 (See http://qa.hq.northscale.net/job/github-couchdb-preview/537/)
        MB-7131 Tweak btree node splits (Revision e30e7397f7d1c30af8ccfcdb10f1e550486aa025)

        Result = SUCCESS
        Farshid Ghods :
        Files :

        • src/couchdb/couch_btree.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #537 (See http://qa.hq.northscale.net/job/github-couchdb-preview/537/ ) MB-7131 Tweak btree node splits (Revision e30e7397f7d1c30af8ccfcdb10f1e550486aa025) Result = SUCCESS Farshid Ghods : Files : src/couchdb/couch_btree.erl
        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Merged: http://review.couchbase.org/#/c/22369/

          People

          • Assignee:
            FilipeManana Filipe Manana (Inactive)
            Reporter:
            FilipeManana Filipe Manana (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes