ns_server is eating up all the connections

OS is REHL. machine is 10.2.1.13

Tried rebalancing 2 REHL nodes.
Rebalance hangs and memcahced is unresponsive to telnet or stats.py

Btw, /opt/NorthScale/1.6.0beta3b/bin/ep_engine/management/stats.py 10.2.1.13:11210 all
Is hung.
/opt/NorthScale/1.6.0beta3b/bin/ep_engine/management/stats.py 10.2.1.14:11210 all
Returns:

[root@localhost ~]# /opt/NorthScale/1.6.0beta3b/bin/ep_engine/management/stats.py 10.2.1.14:11210 all
auth_cmds: 0
auth_errors: 0
bucket_conns: 2
bytes_read: 40231
bytes_written: 4359835
cas_badval: 0
cas_hits: 0
cas_misses: 0
cmd_flush: 0
cmd_get: 0
cmd_set: 0
conn_yields: 0
connection_structures: 12
curr_connections: 12
curr_items: 0
daemon_connections: 10
decr_hits: 0
decr_misses: 0
delete_hits: 0
delete_misses: 0
ep_bg_fetched: 0
ep_commit_time: 0
ep_data_age: 0
ep_data_age_highwat: 0
ep_dbinit: 0
ep_dbname: /opt/NorthScale/1.6.0beta3b/data/ns_1/default
ep_flush_duration: 0
ep_flush_duration_highwat: 0
ep_flusher_state: running
ep_flusher_todo: 0
ep_io_num_read: 1024
ep_io_num_write: 0
ep_io_read_bytes: 0
ep_io_write_bytes: 0
ep_item_commit_failed: 0
ep_item_flush_failed: 0
ep_kv_size: 0
ep_max_data_size: 10485760000
ep_max_txn_size: 50000
ep_mem_high_wat: 7864320000
ep_mem_low_wat: 6291456000
ep_min_data_age: 0
ep_num_eject_failures: 0
ep_num_non_resident: 0
ep_num_pager_runs: 0
ep_num_value_ejects: 0
ep_oom_errors: 0
ep_overhead: 25773280
ep_pending_ops: 0
ep_pending_ops_max: 0
ep_pending_ops_max_duration: 0 usec
ep_pending_ops_total: 0
ep_queue_age_cap: 900
ep_queue_size: 0
ep_storage_age: 0
ep_storage_age_highwat: 0
ep_storage_type: featured
ep_tap_keepalive: 0
ep_too_old: 0
ep_too_young: 0
ep_total_cache_size: 0
ep_total_enqueued: 0
ep_total_persisted: 0
ep_version: 1.6.0beta3b_27_g9544aa2
ep_warmed_up: 0
ep_warmup: true
ep_warmup_thread: complete
ep_warmup_time: 0
get_hits: 0
get_misses: 0
incr_hits: 0
incr_misses: 0
libevent: 1.4.13-stable
limit_maxbytes: 67108864
mem_used: 25773280
pid: 19189
pointer_size: 64
rejected_conns: 0
rusage_system: 0.095985
rusage_user: 0.186971
threads: 4
time: 1282517223
total_connections: 13
uptime: 1127
version: 1.4.4_266_ge579964

From: Sharon Barr
Sent: Sunday, August 22, 2010 3:44 PM
To: Dustin Sallings
Subject: RE: 1.6.0beta3b-41 build is ready

can you check what’s with 10.2.1.13 (it’s REHL).
Once you check it out I can try and reproduce this on REHL.

Sharon

From: Dustin Sallings
Sent: Sunday, August 22, 2010 3:43 PM
To: Sharon Barr
Subject: Re: 1.6.0beta3b-41 build is ready

On Aug 22, 2010, at 15:33, Sharon Barr wrote:

I tried rebalancing without any data and my 10.2.1.13 memcach seem to be on a hung state (no stats on telent, rebalancing is hung - see error below). Can you loging and check it out?

I tried it on 2 windows machines with data and it still failing, I will update the bug.

I see timeouts in your log, but nothing like what we were seeing before.

I can't get stats off your box, either. No idea what to do in this state in Windows. I can't ssh in and attach gdb to see what it's doing. It just isn't responding.

–
Dustin Sallings