Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-12538

Allow multiple reader / writer threads to share the same ForestDB handle

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Unresolved
    • Major
    • techdebt-backlog
    • techdebt-backlog
    • forestdb
    • Security Level: Public
    • None

    Description

      ForestDB currently doesn't support sharing the same database handle among multiple threads. This can cause some inconvenience in some languages such as Go where OS threads are opaque in a language level.

      Marty brought some good comments and suggestions:

      ----------------------------------------------------------------------------------------------------------------------------------------------
      I'm adding additional comments based on having actually used ForestDB a bit.

      1. The "just create a new handle for every thread" sounds like good advice, but in some languages actual OS threads are opaque. For example, in Go it might be common to create thousands of goroutines which want to write to database. These goroutines my actually be executed by only 4 threads, but that fact is hidden from the application, so there is no clean way to only create and safely use 4 ForestDB handles. (yes there may be ways, not yet fully tested, to engineer around this with pooling, but it makes ForestDB considerably harder to use than the competition)

      2. Can two separate threads use the same handle, provided they serialize access to it? This would be an essential property if we are to try and pool the ForestDB handles.

      3. Can readers share a handle concurrently? Can another thread read using same handle as the writer? If so this might allow a different model where all writes are serialized, but we allow for concurrent reads. If yes to both, we can use one handle and just serialize writes. If no to the second question, then we would need to at least 2 handles to accommodate reading while writing. If no to both questions, then using a pool for all readers and writers seems to be the only option.

      4. I think this API difference really hurts in the adoption/evaluation by other key/value database users. I have seen the following pattern which matches my own experience:
      a. User gets a crash and thinks they've found a bug
      b. It is explained to user this isn't a bug, then they think incorrectly think that it cannot be used at all with multiple threads, and they think the project is useless.
      c. Then it is explained to user that it can be used concurrently, but differently than many of the other KV stores. At this point they think it is OK, but very inconvenient.
      d. If they get this far, then they might be a happy user.

      The problem is many people walk away before getting to step d. Right or wrong, people just ASSUME that the handle is thread-safe. I'm not sure any level of documentation can change this core assumption.
      ----------------------------------------------------------------------------------------------------------------------------------------------

      To address the above issues, we will allow multiple threads to share the same ForestDB handle and provide the transparent synchronization, which doesn't require applications to serialize the accesses to the ForestDB handle.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            tai.tran Tai Tran (Inactive)
            chiyoung Chiyoung Seo (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty