Details
-
Bug
-
Resolution: Fixed
-
Major
-
1.8.0
-
Security Level: Public
-
None
-
$uname -a
Linux CentOS-56-64-minimal 2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:31:24 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
$free -m
total used free shared buffers cached
Mem: 24098 12125 11972 0 135 2411
-/+ buffers/cache: 9578 14519
Swap: 24574 0 24574
$./cbstats localhost:11210 all
accepting_conns: 1
auth_cmds: 411810
auth_errors: 0
bucket_active_conns: 1
bucket_conns: 71
bytes_read: 23984196121827
bytes_written: 41798041738415
cas_badval: 3164645
cas_hits: 447032088
cas_misses: 103791
cmd_flush: 0
cmd_get: 65069414539
cmd_set: 4173867648
conn_yields: 2240735
connection_structures: 570
curr_connections: 81
curr_items: 21839182
curr_items_tot: 21839182
daemon_connections: 10
decr_hits: 0
decr_misses: 0
delete_hits: 2357849
delete_misses: 109876589
ep_bg_fetched: 155974
ep_bg_load: 1427658964
ep_bg_load_avg: 9153
ep_bg_max_load: 1597835
ep_bg_max_wait: 127044082
ep_bg_min_load: 5
ep_bg_min_wait: 12
ep_bg_num_samples: 155974
ep_bg_wait: 1792472087975
ep_bg_wait_avg: 11492121
ep_commit_num: 2143861
ep_commit_time: 29
ep_commit_time_total: 22693256
ep_data_age: 1791
ep_data_age_highwat: 8079
ep_db_cleaner_status: complete
ep_db_strategy: multiMTVBDB
ep_dbinit: 0
ep_dbname: /data/membase/data/default-data/default
ep_dbshards: 4
ep_diskqueue_drain: 1496859378
ep_diskqueue_fill: 1496921675
ep_diskqueue_items: 62709
ep_diskqueue_memory: 5016720
ep_diskqueue_pending: 287434364
ep_exp_pager_stime: 3600
ep_expired: 331517089
ep_flush_all: false
ep_flush_duration: 1087
ep_flush_duration_highwat: 5518
ep_flush_duration_total: 22804338
ep_flush_preempts: 6770
ep_flusher_state: running
ep_flusher_todo: 5906
ep_inconsistent_slave_chk: 0
ep_io_num_read: 15406530
ep_io_num_write: 1300314639
ep_io_read_bytes: 3112788722
ep_io_write_bytes: 7023543401129
ep_item_begin_failed: 0
ep_item_commit_failed: 0
ep_item_flush_expired: 309344987
ep_item_flush_failed: 0
ep_items_rm_from_checkpoints: 1117546393
ep_keep_closed_checkpoints: 0
ep_kv_size: 6811396630
ep_latency_arith_cmd: 133770762
ep_latency_get_cmd: 778916369
ep_latency_store_cmd: 462958206
ep_max_data_size: 10485760000
ep_max_txn_size: 1000
ep_mem_high_wat: 7864320000
ep_mem_low_wat: 6291456000
ep_min_data_age: 0
ep_num_active_non_resident: 4093900
ep_num_checkpoint_remover_runs: 4720097
ep_num_eject_failures: 0
ep_num_eject_replicas: 0
ep_num_expiry_pager_runs: 6558
ep_num_non_resident: 4093900
ep_num_not_my_vbuckets: 296617
ep_num_pager_runs: 1
ep_num_value_ejects: 4319110
ep_onlineupdate: false
ep_onlineupdate_revert_add: 0
ep_onlineupdate_revert_delete: 0
ep_onlineupdate_revert_update: 0
ep_oom_errors: 0
ep_overhead: 118345360
ep_pending_ops: 0
ep_pending_ops_max: 0
ep_pending_ops_max_duration: 0
ep_pending_ops_total: 0
ep_queue_age_cap: 900
ep_queue_size: 56811
ep_storage_age: 1788
ep_storage_age_highwat: 20342
ep_storage_type: featured
ep_store_max_concurrency: 1
ep_store_max_readers: 0
ep_store_max_readwrite: 1
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_keepalive: 300
ep_tmp_oom_errors: 0
ep_too_old: 90321337
ep_too_young: 0
ep_total_cache_size: 51059566026
ep_total_del_items: 311560358
ep_total_enqueued: 1742721888
ep_total_new_items: 392087162
ep_total_persisted: 1611874997
ep_uncommitted_items: 978
ep_value_size: 4450861492
ep_vb_total: 512
ep_vbucket_del: 5317
ep_vbucket_del_avg_walltime: 28645
ep_vbucket_del_fail: 0
ep_vbucket_del_max_walltime: 7386666
ep_vbucket_del_total_walltime: 152309494
ep_version: 1.8.0r_78_g3539559
ep_warmed_up: 15249532
ep_warmup: true
ep_warmup_dups: 0
ep_warmup_oom: 0
ep_warmup_thread: complete
ep_warmup_time: 37281275
get_hits: 35331767461
get_misses: 29737647078
incr_hits: 133722528
incr_misses: 13432
libevent: 2.0.11-stable
limit_maxbytes: 67108864
listen_disabled_num: 0
mem_used: 6929741990
pid: 25177
pointer_size: 64
rejected_conns: 0
rusage_system: 1373695.580529
rusage_user: 1061001.812207
tap_checkpoint_end_received: 2571114
tap_checkpoint_end_sent: 2609301
tap_checkpoint_start_received: 2575722
tap_checkpoint_start_sent: 2611861
tap_connect_received: 517
tap_delete_received: 34782238
tap_delete_sent: 33528762
tap_mutation_received: 284121673
tap_mutation_sent: 256131095
tap_opaque_received: 20776
tap_opaque_sent: 8000
threads: 4
time: 1355835351
total_connections: 411855
uptime: 23609388
vb_active_curr_items: 21839182
vb_active_eject: 4319110
vb_active_ht_memory: 100868096
vb_active_itm_memory: 5988964786
vb_active_num: 512
vb_active_num_non_resident: 4093900
vb_active_ops_create: 295325503
vb_active_ops_delete: 281056032
vb_active_ops_reject: 412
vb_active_ops_update: 807785579
vb_active_perc_mem_resident: 81
vb_active_queue_age: 36828698000
vb_active_queue_drain: 1496859378
vb_active_queue_fill: 1496921675
vb_active_queue_memory: 5016720
vb_active_queue_pending: 287434364
vb_active_queue_size: 62709
vb_dead_num: 0
vb_pending_curr_items: 0
vb_pending_eject: 0
vb_pending_ht_memory: 0
vb_pending_itm_memory: 0
vb_pending_num: 0
vb_pending_num_non_resident: 0
vb_pending_ops_create: 0
vb_pending_ops_delete: 0
vb_pending_ops_reject: 0
vb_pending_ops_update: 0
vb_pending_perc_mem_resident: 0
vb_pending_queue_age: 0
vb_pending_queue_drain: 0
vb_pending_queue_fill: 0
vb_pending_queue_memory: 0
vb_pending_queue_pending: 0
vb_pending_queue_size: 0
vb_replica_curr_items: 0
vb_replica_eject: 0
vb_replica_ht_memory: 0
vb_replica_itm_memory: 0
vb_replica_num: 0
vb_replica_num_non_resident: 0
vb_replica_ops_create: 0
vb_replica_ops_delete: 0
vb_replica_ops_reject: 0
vb_replica_ops_update: 0
vb_replica_perc_mem_resident: 0
vb_replica_queue_age: 0
vb_replica_queue_drain: 0
vb_replica_queue_fill: 0
vb_replica_queue_memory: 0
vb_replica_queue_pending: 0
vb_replica_queue_size: 0
version: UNKNOWN
$uname -a Linux CentOS-56-64-minimal 2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:31:24 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux $free -m total used free shared buffers cached Mem: 24098 12125 11972 0 135 2411 -/+ buffers/cache: 9578 14519 Swap: 24574 0 24574 $./cbstats localhost:11210 all accepting_conns: 1 auth_cmds: 411810 auth_errors: 0 bucket_active_conns: 1 bucket_conns: 71 bytes_read: 23984196121827 bytes_written: 41798041738415 cas_badval: 3164645 cas_hits: 447032088 cas_misses: 103791 cmd_flush: 0 cmd_get: 65069414539 cmd_set: 4173867648 conn_yields: 2240735 connection_structures: 570 curr_connections: 81 curr_items: 21839182 curr_items_tot: 21839182 daemon_connections: 10 decr_hits: 0 decr_misses: 0 delete_hits: 2357849 delete_misses: 109876589 ep_bg_fetched: 155974 ep_bg_load: 1427658964 ep_bg_load_avg: 9153 ep_bg_max_load: 1597835 ep_bg_max_wait: 127044082 ep_bg_min_load: 5 ep_bg_min_wait: 12 ep_bg_num_samples: 155974 ep_bg_wait: 1792472087975 ep_bg_wait_avg: 11492121 ep_commit_num: 2143861 ep_commit_time: 29 ep_commit_time_total: 22693256 ep_data_age: 1791 ep_data_age_highwat: 8079 ep_db_cleaner_status: complete ep_db_strategy: multiMTVBDB ep_dbinit: 0 ep_dbname: /data/membase/data/default-data/default ep_dbshards: 4 ep_diskqueue_drain: 1496859378 ep_diskqueue_fill: 1496921675 ep_diskqueue_items: 62709 ep_diskqueue_memory: 5016720 ep_diskqueue_pending: 287434364 ep_exp_pager_stime: 3600 ep_expired: 331517089 ep_flush_all: false ep_flush_duration: 1087 ep_flush_duration_highwat: 5518 ep_flush_duration_total: 22804338 ep_flush_preempts: 6770 ep_flusher_state: running ep_flusher_todo: 5906 ep_inconsistent_slave_chk: 0 ep_io_num_read: 15406530 ep_io_num_write: 1300314639 ep_io_read_bytes: 3112788722 ep_io_write_bytes: 7023543401129 ep_item_begin_failed: 0 ep_item_commit_failed: 0 ep_item_flush_expired: 309344987 ep_item_flush_failed: 0 ep_items_rm_from_checkpoints: 1117546393 ep_keep_closed_checkpoints: 0 ep_kv_size: 6811396630 ep_latency_arith_cmd: 133770762 ep_latency_get_cmd: 778916369 ep_latency_store_cmd: 462958206 ep_max_data_size: 10485760000 ep_max_txn_size: 1000 ep_mem_high_wat: 7864320000 ep_mem_low_wat: 6291456000 ep_min_data_age: 0 ep_num_active_non_resident: 4093900 ep_num_checkpoint_remover_runs: 4720097 ep_num_eject_failures: 0 ep_num_eject_replicas: 0 ep_num_expiry_pager_runs: 6558 ep_num_non_resident: 4093900 ep_num_not_my_vbuckets: 296617 ep_num_pager_runs: 1 ep_num_value_ejects: 4319110 ep_onlineupdate: false ep_onlineupdate_revert_add: 0 ep_onlineupdate_revert_delete: 0 ep_onlineupdate_revert_update: 0 ep_oom_errors: 0 ep_overhead: 118345360 ep_pending_ops: 0 ep_pending_ops_max: 0 ep_pending_ops_max_duration: 0 ep_pending_ops_total: 0 ep_queue_age_cap: 900 ep_queue_size: 56811 ep_storage_age: 1788 ep_storage_age_highwat: 20342 ep_storage_type: featured ep_store_max_concurrency: 1 ep_store_max_readers: 0 ep_store_max_readwrite: 1 ep_tap_bg_fetch_requeued: 0 ep_tap_bg_fetched: 0 ep_tap_keepalive: 300 ep_tmp_oom_errors: 0 ep_too_old: 90321337 ep_too_young: 0 ep_total_cache_size: 51059566026 ep_total_del_items: 311560358 ep_total_enqueued: 1742721888 ep_total_new_items: 392087162 ep_total_persisted: 1611874997 ep_uncommitted_items: 978 ep_value_size: 4450861492 ep_vb_total: 512 ep_vbucket_del: 5317 ep_vbucket_del_avg_walltime: 28645 ep_vbucket_del_fail: 0 ep_vbucket_del_max_walltime: 7386666 ep_vbucket_del_total_walltime: 152309494 ep_version: 1.8.0r_78_g3539559 ep_warmed_up: 15249532 ep_warmup: true ep_warmup_dups: 0 ep_warmup_oom: 0 ep_warmup_thread: complete ep_warmup_time: 37281275 get_hits: 35331767461 get_misses: 29737647078 incr_hits: 133722528 incr_misses: 13432 libevent: 2.0.11-stable limit_maxbytes: 67108864 listen_disabled_num: 0 mem_used: 6929741990 pid: 25177 pointer_size: 64 rejected_conns: 0 rusage_system: 1373695.580529 rusage_user: 1061001.812207 tap_checkpoint_end_received: 2571114 tap_checkpoint_end_sent: 2609301 tap_checkpoint_start_received: 2575722 tap_checkpoint_start_sent: 2611861 tap_connect_received: 517 tap_delete_received: 34782238 tap_delete_sent: 33528762 tap_mutation_received: 284121673 tap_mutation_sent: 256131095 tap_opaque_received: 20776 tap_opaque_sent: 8000 threads: 4 time: 1355835351 total_connections: 411855 uptime: 23609388 vb_active_curr_items: 21839182 vb_active_eject: 4319110 vb_active_ht_memory: 100868096 vb_active_itm_memory: 5988964786 vb_active_num: 512 vb_active_num_non_resident: 4093900 vb_active_ops_create: 295325503 vb_active_ops_delete: 281056032 vb_active_ops_reject: 412 vb_active_ops_update: 807785579 vb_active_perc_mem_resident: 81 vb_active_queue_age: 36828698000 vb_active_queue_drain: 1496859378 vb_active_queue_fill: 1496921675 vb_active_queue_memory: 5016720 vb_active_queue_pending: 287434364 vb_active_queue_size: 62709 vb_dead_num: 0 vb_pending_curr_items: 0 vb_pending_eject: 0 vb_pending_ht_memory: 0 vb_pending_itm_memory: 0 vb_pending_num: 0 vb_pending_num_non_resident: 0 vb_pending_ops_create: 0 vb_pending_ops_delete: 0 vb_pending_ops_reject: 0 vb_pending_ops_update: 0 vb_pending_perc_mem_resident: 0 vb_pending_queue_age: 0 vb_pending_queue_drain: 0 vb_pending_queue_fill: 0 vb_pending_queue_memory: 0 vb_pending_queue_pending: 0 vb_pending_queue_size: 0 vb_replica_curr_items: 0 vb_replica_eject: 0 vb_replica_ht_memory: 0 vb_replica_itm_memory: 0 vb_replica_num: 0 vb_replica_num_non_resident: 0 vb_replica_ops_create: 0 vb_replica_ops_delete: 0 vb_replica_ops_reject: 0 vb_replica_ops_update: 0 vb_replica_perc_mem_resident: 0 vb_replica_queue_age: 0 vb_replica_queue_drain: 0 vb_replica_queue_fill: 0 vb_replica_queue_memory: 0 vb_replica_queue_pending: 0 vb_replica_queue_size: 0 version: UNKNOWN
-
Untriaged
Description
Couchbase will start evicting documents when server memory filled with system disk cache, even when couchbase memory usage is well below bucket quota.
To prevent this situation most people dropping system disk cache in cron task like this:
echo 3 > /proc/sys/vm/drop_caches
Also i don't understand why disk cache created, because Couchbase have no reads from disk(only writes if it set to permanent, like at my servers).
Discussion in Couchbase Google group: https://groups.google.com/forum/#!topic/couchbase/H52nP3TualU/discussion
Screenshots of Web UI stats:
http://gyazo.com/acd03396c42784069cba962dedfe2e6a
http://gyazo.com/be4897fadfa5fa1c90cac550591e1d39
http://gyazo.com/3900dfeffc90c2e42f5ce7b7d0fd0f46
http://gyazo.com/c434b8ead2a7207df9e6a6936ce632ac
http://gyazo.com/63dd9b44d8e2fe7812ea423a1a748276
http://gyazo.com/e66503094d3285d95f7a40cc4d8c83fd
Other data on screenshots is the system disk cache created by Couchbase operations for one day.