Details
-
Bug
-
Resolution: Done
-
Major
-
Cheshire-Cat
-
7.0.0-4310
-
Untriaged
-
1
-
Unknown
Description
On running 3K Index Test with storage as MOI : we are seeing the following error
2021-01-26T22:06:00.783-08:00 [Error] MemDBSlice Slice Id 0, IndexInstId 1729060521352474433, PartitionId 0 failed to create ondisk snapshot /data/@2i/bucket-1_index23-2_1729060521352474433_0.index/snapshot.2021-01-26.220552.684 (error=open /data/@2i/bucket-1_index23-2_1729060521352474433_0.index/.tmp/delta/shard-37: too many open files)
|
2021-01-26T22:06:00.792-08:00 [Error] MemDBSlice Slice Id 0, IndexInstId 2489186406576490176, PartitionId 0 failed to create ondisk snapshot /data/@2i/bucket-1_index252-7_2489186406576490176_0.index/snapshot.2021-01-26.220552.697 (error=open /data/@2i/bucket-1_index252-7_2489186406576490176_0.index/.tmp/delta/shard-19: too many open files)
|
2021-01-26T22:06:00.882-08:00 [Error] MemDBSlice Slice Id 0, IndexInstId 9486440994981139476, PartitionId 0 failed to create ondisk snapshot /data/@2i/bucket-1_index230-9_9486440994981139476_0.index/snapshot.2021-01-26.220552.684 (error=open /data/@2i/bucket-1_index230-9_9486440994981139476_0.index/.tmp/delta/shard-57: too many open files)
|
2021-01-26T22:06:00.890-08:00 [Error] MemDBSlice Slice Id 0, IndexInstId 16104320503682095074, PartitionId 0 failed to create ondisk snapshot /data/@2i/bucket-1_index47-5_16104320503682095074_0.index/snapshot.2021-01-26.220552.674 (error=open /data/@2i/bucket-1_index47-5_16104320503682095074_0.index/.tmp/delta/shard-48: too many open files)
|
2021-01-26T22:06:00.897-08:00 [Error] MemDBSlice Slice Id 0, IndexInstId 14244156889870169334, PartitionId 0 failed to create ondisk snapshot /data/@2i/bucket-1_index103-4_14244156889870169334_0.index/snapshot.2021-01-26.220552.685 (error=open /data/@2i/bucket-1_index103-4_14244156889870169334_0.index/.tmp/delta/shard-75: too many open files)
|
2021-01-26T22:06:00.977-08:00 [Error] MemDBSlice Slice Id 0, IndexInstId 12691817475769855061, PartitionId 0 failed to create ondisk snapshot /data/@2i/bucket-1_index295-8_12691817475769855061_0.index/snapshot.2021-01-26.220552.699 (error=open /data/@2i/bucket-1_index295-8_12691817475769855061_0.index/.tmp/delta/shard-15: too many open files)
|
The run : http://perf.jenkins.couchbase.com/view/GSI/job/hemera/775/console
The FD limit on indexer node was increased from 1024 to 200000, and we still see the same issue in the indexer logs.
couchbase user limits
|
su couchbase -s /bin/sh -c "ulimit -a"
|
==============================================================================
|
core file size (blocks, -c) 0
|
data seg size (kbytes, -d) unlimited
|
scheduling priority (-e) 0
|
file size (blocks, -f) unlimited
|
pending signals (-i) 2055971
|
max locked memory (kbytes, -l) 64
|
max memory size (kbytes, -m) unlimited
|
open files (-n) 200000
|
pipe size (512 bytes, -p) 8
|
POSIX message queues (bytes, -q) 819200
|
real-time priority (-r) 0
|
stack size (kbytes, -s) 8192
|
cpu time (seconds, -t) unlimited
|
max user processes (-u) 4096
|
virtual memory (kbytes, -v) unlimited
|
file locks (-x) unlimited
|
==============================================================================
|