Details
-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
4.0.0
-
Security Level: Public
-
None
-
Untriaged
-
Ubuntu 64-bit
-
Unknown
Description
I believe there are 2 different variants of FD leak showing on my test cluster running KV, indexer and n1ql on same node
{markdown}- Server is sitting idle for most of the time, with very light KV workload(<10 ops/sec)
- 1st type of FD leak is because of half opened connections, climbing up at the rate of 8/sec.
root@galleon:~# echo indexer | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p | grep "can't identify protocol" | wc -l |
29678
|
|
root@galleon:~# echo indexer | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p | grep "can't identify protocol" | head -10 |
indexer 12136 couchbase 14u sock 0,7 0t0 91776188 can't identify protocol
|
indexer 12136 couchbase 17u sock 0,7 0t0 91862154 can't identify protocol
|
indexer 12136 couchbase 18u sock 0,7 0t0 91757759 can't identify protocol
|
indexer 12136 couchbase 19u sock 0,7 0t0 91764280 can't identify protocol
|
indexer 12136 couchbase 20u sock 0,7 0t0 91766279 can't identify protocol
|
indexer 12136 couchbase 21u sock 0,7 0t0 91849218 can't identify protocol
|
indexer 12136 couchbase 22u sock 0,7 0t0 91775669 can't identify protocol
|
indexer 12136 couchbase 26u sock 0,7 0t0 91862155 can't identify protocol
|
indexer 12136 couchbase 27u sock 0,7 0t0 91764284 can't identify protocol
|
indexer 12136 couchbase 28u sock 0,7 0t0 91775061 can't identify protocol
|
- 2nd kind of FD leak is on port 8091(Looks like Sarath identified cause for it)
root@galleon:~# echo indexer | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p | grep ":8091" | wc -l |
1496
|
|
root@galleon:~# echo indexer | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p | grep ":8091" | head -10 |
indexer 12136 couchbase 4u IPv4 91743124 0t0 TCP 127.0.0.1:54993->127.0.0.1:8091 (ESTABLISHED)
|
indexer 12136 couchbase 5u IPv4 91776180 0t0 TCP 127.0.0.1:54994->127.0.0.1:8091 (ESTABLISHED)
|
indexer 12136 couchbase 8u IPv4 91861179 0t0 TCP 127.0.0.1:54995->127.0.0.1:8091 (ESTABLISHED)
|
indexer 12136 couchbase *802u IPv4 92141035 0t0 TCP 127.0.0.1:41559->127.0.0.1:8091 (ESTABLISHED)
|
indexer 12136 couchbase *025u IPv4 92902585 0t0 TCP 127.0.0.1:44266->127.0.0.1:8091 (CLOSE_WAIT)
|
indexer 12136 couchbase *088u IPv4 92898572 0t0 TCP 127.0.0.1:44272->127.0.0.1:8091 (CLOSE_WAIT)
|
indexer 12136 couchbase *112u IPv4 92891640 0t0 TCP 127.0.0.1:44269->127.0.0.1:8091 (CLOSE_WAIT)
|
indexer 12136 couchbase *126u IPv4 92888529 0t0 TCP 127.0.0.1:44270->127.0.0.1:8091 (CLOSE_WAIT)
|
indexer 12136 couchbase *129u IPv4 92889705 0t0 TCP 127.0.0.1:44273->127.0.0.1:8091 (CLOSE_WAIT)
|
indexer 12136 couchbase *149u IPv4 92878542 0t0 TCP 127.0.0.1:44294->127.0.0.1:8091 (CLOSE_WAIT)
|
I have encountered this issue twice within a matter in past 3 days. cbcollect_info from yesterday: https://s3.amazonaws.com/customers.couchbase.com/couchbase/indexer-fd_leak.zip