Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
3.1.0
-
Security Level: Public
-
[info] OS Name : Linux 2.6.32-573.1.1.el6.x86_64
[info] OS Version : Red Hat Enterprise Linux Server release 6.7 (Santiago)
[info] CB Version : 3.1.0-1776-rel-enterprise [
info] CB Packages : couchbase-server-3.1.0-1776.x86_64
[warn] CB Uptime : ??? (Attribute not found). Cannot check.
[info] System Uptime : 33 min [info] Architecture : x86_64[info] OS Name : Linux 2.6.32-573.1.1.el6.x86_64 [info] OS Version : Red Hat Enterprise Linux Server release 6.7 (Santiago) [info] CB Version : 3.1.0-1776-rel-enterprise [ info] CB Packages : couchbase-server-3.1.0-1776.x86_64 [warn] CB Uptime : ??? (Attribute not found). Cannot check. [info] System Uptime : 33 min [info] Architecture : x86_64
-
Untriaged
-
Centos 64-bit
-
Unknown
Description
Rebalance kept failing on one node.
There are many memcached connections which exceed the max limits 30,000.
The memcached logs also show the same:
Thu Aug 13 18:28:49.832053 GMT 3: Too many open connections
|
Thu Aug 13 18:28:49.832077 GMT 3: Too many open connections
|
Thu Aug 13 18:28:49.832106 GMT 3: Too many open connections
|
Thu Aug 13 18:28:49.832128 GMT 3: Too many open connections
|
Thu Aug 13 18:28:49.832146 GMT 3: Too many open connections
|
Thu Aug 13 18:28:49.832171 GMT 3: Too many open connections
|
Thu Aug 13 18:28:49.832186 GMT 3: Too many open connections
|
Also from lsof output, there are many below entries:
memcached 5541 couchbase 1422u sock 0,6 0t0 749862 can't identify protocol
|
memcached 5541 couchbase 1423u sock 0,6 0t0 773067 can't identify protocol
|
memcached 5541 couchbase 1424u sock 0,6 0t0 629256 can't identify protocol
|
memcached 5541 couchbase 1425u sock 0,6 0t0 826786 can't identify protocol
|
memcached 5541 couchbase 1426u sock 0,6 0t0 653344 can't identify protocol
|
memcached 5541 couchbase 1427u sock 0,6 0t0 677622 can't identify protocol
|
memcached 5541 couchbase 1428u sock 0,6 0t0 151754 can't identify protocol
|
memcached 5541 couchbase 1429u sock 0,6 0t0 637362 can't identify protocol
|
memcached 5541 couchbase 1430u sock 0,6 0t0 53756963 can't identify protocol
|
memcached 5541 couchbase 1431u sock 0,6 0t0 269680 can't identify protocol
|
memcached 5541 couchbase 1432u sock 0,6 0t0 490188 can't identify protocol
|
After some research, those "can't identify protocol" messages indicates that the system reach the open file limit for couchbase user whose ulimit of open file is set to 40960.
I think there is a possible FD leak in memcached.