Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-23665

[FTS] moss store potential data loss with parallel store.Persist() invocations

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Minor
    • 5.0.0
    • 5.0.0
    • moss
    • None
    • Untriaged
    • Unknown

    Description

      Say we have 2 go routines writing data into mossStore,
      go routine 1: ExecuteBatch() with Segment X, takes snapshot ( X ) & calls store.Persist( X )
      go routine 2: ExecuteBatch() with Segment Y, takes snapshot (X+Y) & calls store.Persist(X+Y)

      Now go routine 1's Persist ( X ) is called first grabs the store lock, builds new store footer and releases the store lock here

      https://github.com/couchbase/moss/blob/master/store.go#L136

      then goes ahead and issues I/O to actually persist the segment ( X ) 

      go routine 2 calls Persist (X+Y) grabs the lock released by routine 1, builds store footer footer and is now persisting segment ( X + Y).

      Now due to I/O race, go routine 2's write somehow completes first and it manages to grab the second lock here

      https://github.com/couchbase/moss/blob/master/store.go#L159

      Now the footer has X + Y and everything is good.. until...

      go routine 1 completes its I/O and writes out its footer which only has segment X.

      This results in data loss of segment Y in case there is a shutdown or crash immediately after as far as go routine 2 is concerned.

      Simple fix might just be to hold on to the store lock until the footer is persisted, but need to evaluate the side effects on things like snapshots etc.

      thanks

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sundar Sundar Sridharan (Inactive)
            sundar Sundar Sridharan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty