RAM usage doubled after rebalancing

There are few bugs related to memory / water mark on a cluster with more than one node, and I believe that few of them could derive from this core issue.

RAM usage is not calculated correctly after adding a second node to a cluster. In some cases in doubled.

Below is the Usage of 1 and 2 nodes, including stats.py from 10.2.1.13 (RAM usage went up from 1.51GB on default bucket to 2.99GB

Ram Usage of a given bucket

1 Node:
Bucket Name Bucket Type Bucket Resides On RAM Usage/Quota Disk Usage/Quota
ascii-11213
Membase 1 server 554MB/1000MB 1.58GB/1024MB
default
Membase 1 server 1.51GB/2.92GB 1.58GB/4GB

[root@localhost ~]# /opt/NorthScale/1.6.0beta3b/bin/ep_engine/management/stats.py 10.2.1.13:11213 all
auth_cmds: 1038
auth_errors: 0
bucket_conns: 7
bytes_read: 806231584
bytes_written: 8997522
cas_badval: 0
cas_hits: 0
cas_misses: 0
cmd_flush: 0
cmd_get: 0
cmd_set: 192847
conn_yields: 2179
connection_structures: 32
curr_connections: 22
curr_items: 96424
daemon_connections: 10
decr_hits: 0
decr_misses: 0
delete_hits: 0
delete_misses: 0
ep_bg_fetched: 369785
ep_bg_num_samples: 369785
ep_commit_time: 0
ep_data_age: 2
ep_data_age_highwat: 5
ep_dbinit: 0
ep_dbname: /opt/NorthScale/1.6.0beta3b/data/ns_1/ascii-11213
ep_flush_duration: 0
ep_flush_duration_highwat: 2
ep_flusher_state: running
ep_flusher_todo: 10
ep_io_num_read: 369785
ep_io_num_write: 194493
ep_io_read_bytes: 760019985
ep_io_write_bytes: 396386973
ep_item_commit_failed: 0
ep_item_flush_failed: 0
ep_kv_size: 540030384
ep_max_data_size: 1048576000
ep_max_txn_size: 50000
ep_mem_high_wat: 786432000
ep_mem_low_wat: 629145600
ep_min_data_age: 0
ep_num_eject_failures: 0
ep_num_non_resident: 26537
ep_num_pager_runs: 1
ep_num_value_ejects: 53037
ep_oom_errors: 105108
ep_overhead: 41324253
ep_pending_ops: 0
ep_pending_ops_max: 0
ep_pending_ops_total: 0
ep_queue_age_cap: 900
ep_queue_size: 90689
ep_storage_age: 0
ep_storage_age_highwat: 3
ep_storage_type: featured
ep_tap_keepalive: 4645872405673059
ep_too_old: 0
ep_too_young: 0
ep_total_cache_size: 594291922
ep_total_enqueued: 285193
ep_total_persisted: 192847
ep_version: 1.6.0beta3b_31_gdd8e664
ep_warmed_up: 0
ep_warmup: true
ep_warmup_thread: complete
ep_warmup_time: 0
get_hits: 0
get_misses: 0
incr_hits: 0
incr_misses: 0
libevent: 1.4.13-stable
limit_maxbytes: 67108864
mem_used: 581354637
pid: 23373
pointer_size: 64
rejected_conns: 0
rusage_system: 439.518183
rusage_user: 507.053916
tap_connect_received: 1546
tap_mutation_received: 947239
tap_mutation_sent: 917748
tap_vbucket_set_received: 4096
tap_vbucket_set_sent: 3072
threads: 4
time: 1282787023
total_connections: 3619
uptime: 670
version: 1.4.4_271_g1d62667

2 Node (After rebalancing):
Bucket Name Bucket Type Bucket Resides On RAM Usage/Quota Disk Usage/Quota
ascii-11213
Membase 2 servers 1054MB/1000MB 3.57GB/1024MB
default
Membase 2 servers 2.99GB/2.92GB 3.57GB/4GB

[root@localhost ~]# /opt/NorthScale/1.6.0beta3b/bin/ep_engine/management/stats.py 10.2.1.13:11213 all
auth_cmds: 2075
auth_errors: 0
bucket_conns: 19
bytes_read: 1011009082
bytes_written: 14220221
cas_badval: 0
cas_hits: 0
cas_misses: 0
cmd_flush: 0
cmd_get: 2393
cmd_set: 194024
conn_yields: 2473
connection_structures: 52
curr_connections: 49
curr_items: 127107
daemon_connections: 20
decr_hits: 0
decr_misses: 0
delete_hits: 0
delete_misses: 0
ep_bg_fetched: 369785
ep_bg_num_samples: 369785
ep_commit_time: 8
ep_data_age: 2
ep_data_age_highwat: 5
ep_dbinit: 0
ep_dbname: 0.000000
ep_flush_duration: 413
ep_flush_duration_highwat: 413
ep_flusher_state: running
ep_flusher_todo: 124547
ep_io_num_read: 563656
ep_io_num_write: 194911
ep_io_read_bytes: 1156406958
ep_io_write_bytes: 396386973
ep_item_commit_failed: 0
ep_item_flush_failed: 0
ep_kv_size: 1013459811
ep_max_data_size: 1048576000
ep_max_txn_size: 100000
ep_mem_high_wat: 786432000
ep_mem_low_wat: 629145600
ep_min_data_age: 0
ep_num_eject_failures: 11486
ep_num_non_resident: 26537
ep_num_pager_runs: 2
ep_num_value_ejects: 53037
ep_oom_errors: 171989
ep_overhead: 68135372
ep_pending_ops: 0
ep_pending_ops_max: 0
ep_pending_ops_total: 0
ep_queue_age_cap: 1800
ep_queue_size: 1648
ep_storage_age: 0
ep_storage_age_highwat: 3
ep_storage_type: featured
ep_tap_keepalive: 4645872405673059
ep_too_old: 0
ep_too_young: 0
ep_total_cache_size: 1067721349
ep_total_enqueued: 321108
ep_total_persisted: 192847
ep_version: 3.200000
ep_warmed_up: 192847
ep_warmup: true
ep_warmup_thread: complete
ep_warmup_time: 5
get_hits: 239
get_misses: 2154
incr_hits: 0
incr_misses: 0
libevent: 2.800000
limit_maxbytes: 67108864
mem_used: 1081595183
pid: 23373
pointer_size: 64
rejected_conns: 0
rusage_system: 919.934143
rusage_user: 1098.520996
tap_connect_received: 2576
tap_mutation_received: 1309897
tap_mutation_sent: 1280406
tap_vbucket_set_received: 6144
tap_vbucket_set_sent: 5120
threads: 8
time: 1282787340
total_connections: 5699
uptime: 987
version: 1.400000