Details
-
Improvement
-
Resolution: Won't Do
-
Major
-
4.0.0
-
Security Level: Public
-
None
Description
Currently, secondary indexing uses fdb_set_kv API for insertions into ForestDB.
fdb_set_kv API call fdb_doc_create for every new document inserted.
https://github.com/couchbase/forestdb/blob/master/src/api_wrapper.cc#L96
the fdb_doc_create call allocates new memory on the heap for each insertion.
https://github.com/couchbase/forestdb/blob/master/src/forestdb.cc#L1680
This takes additional CPU cycles for memory management calls and increases memory fragmentation and also additional memory usage until the fdb_doc structure is freed.
Using the fdb_set API instead of fdb_set_kv API for insert operations and re-using a fdb_doc structure with sufficiently large key size will yield better performance.For any keys that are larger than this pre-allocated fdb_doc structure
can handle a new fdb_doc_create may be issued but such keys are expected to be few or a configuration may be provided to the user to determine the key size of this pre-allocated fdb_doc structure.
Expect improvement in fragmentation to be higher than in actual index build times.