Details
-
Bug
-
Resolution: Won't Fix
-
Major
-
None
-
None
-
Security Level: Public
-
None
-
elasticsearch 0.22.2
CentOS release 5.4 (Final)
4 cpu
4GM mem
Description
I have 3 couchbase nodes replicating to 1 elastic search node
using mcsoda to load 1 million documents to couchbase:
ib/perf_engines/mcsoda.py 10.1.3.78:11211 doc-cache=0 doc-gen=0 max-items=1000000 ratio-sets=1.0 ratio-creates=1.0 threads=4
at about 300k documents started to get errors from elasticsearch (see below and attached file for node named 'Hazard')
It's possible error is coming from too many open files:
Caused by: java.io.FileNotFoundException: /home/tommie/elasticsearch-0.20.2/data/elasticsearch/nodes/0/indices/default/0/index/_37l.fdt (Too many open files)
es started as root: sudo elasticsearch -f
2013-01-17 07:42:18,719][DEBUG][action.bulk ] [Hazard] [default][0] failed to execute bulk item (index) index {[default][couchbaseDocument][912f1b8d80ecabdc], source[{"doc":
{"key":"912f1b8d80ecabdc","key_num":175481,"name":"912f abd","email":"912f@f1.com","city":"1b8","country":"d8","realm":"0e","coins":371.67,"category":1,"achievements":[48,112,112,252,4,147,9],"body":"xc4ca4238a0b923820dcc509a6f75849b"},"meta":{"id":"912f1b8d80ecabdc","rev":"2-00433642aa647bea0000000000000000","expiration":0,"flags":0}}]}
org.elasticsearch.index.engine.IndexFailedEngineException: [default][0] Index failed for couchbaseDocument#912f1b8d80ecabdc
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:494)
at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:330)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:533)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:431)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException: /home/tommie/elasticsearch-0.20.2/data/elasticsearch/nodes/0/indices/default/0/index/_37l.fdt (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at org.apache.lucene.store.FSDirectory$FSIndexOutocumentsWriter.finishDocument(DocumentsWriter.java:1043)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060)
at org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:577)
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:489)
... 7 more