Details
-
Story
-
Resolution: Fixed
-
Major
-
1.6.4
-
Security Level: Public
-
None
Description
If all the memory is taken up by metadata (which is the case here) it seems like we won't have any room to bring items in from disk, let alone store new data. I "think" we should set a separate max memory limit just for metadata so that the server can't get into this state. It can still respond with OOM if it can't add any new items, but it should reserve some space for data. If that limit is then configurable, we can tune it appropriately for the item size and number of items. We can also then alert the user that they have filled up the "item" space and need to take some action to increase space but it won't completely hose the system.
Stats showing this problem:
$ ./stats localhost:11210 all
auth_cmds: 1
auth_errors: 0
bucket_conns: 3
bytes_read: 4111710999
bytes_written: 451661495
cas_badval: 0
cas_hits: 0
cas_misses: 0
cmd_flush: 1
cmd_get: 1162293
cmd_set: 2325506
conn_yields: 0
connection_structures: 8
curr_connections: 8
curr_items: 2324571
curr_items_tot: 2324571
daemon_connections: 5
decr_hits: 0
decr_misses: 0
delete_hits: 0
delete_misses: 0
ep_bg_fetched: 0
ep_commit_num: 10398
ep_commit_time: 0
ep_commit_time_total: 2144
ep_data_age: 2
ep_data_age_highwat: 12
ep_dbinit: 0
ep_dbname: /home/membase/1.6.0.1/data/ns_1/default
ep_dbshards: 4
ep_expired: 0
ep_flush_duration: 0
ep_flush_duration_highwat: 8
ep_flush_duration_total: 2175
ep_flush_preempts: 0
ep_flusher_state: running
ep_flusher_todo: 0
ep_io_num_read: 0
ep_io_num_write: 2324571
ep_io_read_bytes: 0
ep_io_write_bytes: 3937810793
ep_item_commit_failed: 0
ep_item_flush_expired: 0
ep_item_flush_failed: 0
ep_kv_size: 255507713
ep_max_data_size: 268435456
ep_max_txn_size: 250000
ep_mem_high_wat: 201326592
ep_mem_low_wat: 161061273
ep_min_data_age: 0
ep_num_eject_failures: 639999137
ep_num_expiry_pager_runs: 4
ep_num_non_resident: 2324571
ep_num_not_my_vbuckets: 0
ep_num_pager_runs: 939
ep_num_value_ejects: 2324571
ep_oom_errors: 228
ep_overhead: 12927624
ep_pending_ops: 0
ep_pending_ops_max: 0
ep_pending_ops_max_duration: 0
ep_pending_ops_total: 0
ep_queue_age_cap: 900
ep_queue_size: 0
ep_storage_age: 0
ep_storage_age_highwat: 9
ep_storage_type: featured
ep_tap_keepalive: 0
ep_tmp_oom_errors: 707
ep_too_old: 0
ep_too_young: 0
ep_total_cache_size: 4076044533
ep_total_del_items: 0
ep_total_enqueued: 2324572
ep_total_new_items: 2324571
ep_total_persisted: 2324571
ep_vbucket_del: 0
ep_vbucket_del_fail: 0
ep_version: 1.6.0_10_g3b4878a
ep_warmed_up: 0
ep_warmup: true
ep_warmup_dups: 0
ep_warmup_oom: 0
ep_warmup_thread: complete
ep_warmup_time: 0
get_hits: 0
get_misses: 1162293
incr_hits: 0
incr_misses: 0
libevent: 2.0.8-rc
limit_maxbytes: 67108864
mem_used: 268435337
pid: 6304
pointer_size: 32
rejected_conns: 0
rusage_system: 50.975250
rusage_user: 502.402623
threads: 4
time: 1289588201
total_connections: 14413
uptime: 17354
version: 1.4.4_298_g250909b