Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-2816

As a Membase user, I should be able to fill up my memory space with metadata while still maintaining proper operation of the server

    XMLWordPrintable

Details

    • Story
    • Resolution: Fixed
    • Major
    • techdebt-backlog
    • 1.6.4
    • couchbase-bucket
    • Security Level: Public
    • None

    Description

      If all the memory is taken up by metadata (which is the case here) it seems like we won't have any room to bring items in from disk, let alone store new data. I "think" we should set a separate max memory limit just for metadata so that the server can't get into this state. It can still respond with OOM if it can't add any new items, but it should reserve some space for data. If that limit is then configurable, we can tune it appropriately for the item size and number of items. We can also then alert the user that they have filled up the "item" space and need to take some action to increase space but it won't completely hose the system.

      Stats showing this problem:
      $ ./stats localhost:11210 all
      auth_cmds: 1
      auth_errors: 0
      bucket_conns: 3
      bytes_read: 4111710999
      bytes_written: 451661495
      cas_badval: 0
      cas_hits: 0
      cas_misses: 0
      cmd_flush: 1
      cmd_get: 1162293
      cmd_set: 2325506
      conn_yields: 0
      connection_structures: 8
      curr_connections: 8
      curr_items: 2324571
      curr_items_tot: 2324571
      daemon_connections: 5
      decr_hits: 0
      decr_misses: 0
      delete_hits: 0
      delete_misses: 0
      ep_bg_fetched: 0
      ep_commit_num: 10398
      ep_commit_time: 0
      ep_commit_time_total: 2144
      ep_data_age: 2
      ep_data_age_highwat: 12
      ep_dbinit: 0
      ep_dbname: /home/membase/1.6.0.1/data/ns_1/default
      ep_dbshards: 4
      ep_expired: 0
      ep_flush_duration: 0
      ep_flush_duration_highwat: 8
      ep_flush_duration_total: 2175
      ep_flush_preempts: 0
      ep_flusher_state: running
      ep_flusher_todo: 0
      ep_io_num_read: 0
      ep_io_num_write: 2324571
      ep_io_read_bytes: 0
      ep_io_write_bytes: 3937810793
      ep_item_commit_failed: 0
      ep_item_flush_expired: 0
      ep_item_flush_failed: 0
      ep_kv_size: 255507713
      ep_max_data_size: 268435456
      ep_max_txn_size: 250000
      ep_mem_high_wat: 201326592
      ep_mem_low_wat: 161061273
      ep_min_data_age: 0
      ep_num_eject_failures: 639999137
      ep_num_expiry_pager_runs: 4
      ep_num_non_resident: 2324571
      ep_num_not_my_vbuckets: 0
      ep_num_pager_runs: 939
      ep_num_value_ejects: 2324571
      ep_oom_errors: 228
      ep_overhead: 12927624
      ep_pending_ops: 0
      ep_pending_ops_max: 0
      ep_pending_ops_max_duration: 0
      ep_pending_ops_total: 0
      ep_queue_age_cap: 900
      ep_queue_size: 0
      ep_storage_age: 0
      ep_storage_age_highwat: 9
      ep_storage_type: featured
      ep_tap_keepalive: 0
      ep_tmp_oom_errors: 707
      ep_too_old: 0
      ep_too_young: 0
      ep_total_cache_size: 4076044533
      ep_total_del_items: 0
      ep_total_enqueued: 2324572
      ep_total_new_items: 2324571
      ep_total_persisted: 2324571
      ep_vbucket_del: 0
      ep_vbucket_del_fail: 0
      ep_version: 1.6.0_10_g3b4878a
      ep_warmed_up: 0
      ep_warmup: true
      ep_warmup_dups: 0
      ep_warmup_oom: 0
      ep_warmup_thread: complete
      ep_warmup_time: 0
      get_hits: 0
      get_misses: 1162293
      incr_hits: 0
      incr_misses: 0
      libevent: 2.0.8-rc
      limit_maxbytes: 67108864
      mem_used: 268435337
      pid: 6304
      pointer_size: 32
      rejected_conns: 0
      rusage_system: 50.975250
      rusage_user: 502.402623
      threads: 4
      time: 1289588201
      total_connections: 14413
      uptime: 17354
      version: 1.4.4_298_g250909b

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            dipti Dipti Borkar (Inactive)
            perry Perry Krug
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty