Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45622

[CBM] Improve resilience to power outage/kill -9 scenarios

    XMLWordPrintable

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 6.5.0, 6.0.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.5.1, 6.0.5, 6.6.0, 6.6.1, 6.6.2, 6.5.2, 7.0.0, 6.6.3, 7.1.0, 7.0.1
    • feature-backlog
    • tools
    • None
    • 1

    Description

      What's the issue?
      Although we do have logic to handle unexpected failures such as power outages and a 'kill -9', it's currently built of unsafe assumptions, these assumptions are:

      1. That SQLite will be able to recover from a power outage (by default, this should be the case, however, we disable journals and syncing)
      2. The the 'RiftBufferedWriter' will be able to recover from a power outage (we use the 'sync_file_range' syscall as a performance optimization; this isn't safe on some filesystems as file metadata will not be written out as it would with 'fdatasync').

      Example
      Some of our testing will prematurely 'kill -9' 'cbbackupmgr' in an effort to test resume support, this has lead to situations where an invalid/corrupt SQLite file is detected. See this case where one of our tests has failed due to a 'database disk image malformed'.

      What's the fix?
      Ideally, we should better handle these situations where possible:

      1. Move away from using the truncate-overwrite pattern (MB-46878)
      2. Enable syncing for SQLite (potentially enable journals, although we need to consider that not all journal types are supported on NFS)
      3. Periodically sync using 'fsync' or 'fdatasync' in the 'RiftBufferedWriter'

      Attachments

        1. cbbackupmgr-collectinfo-backups-2021-04-13T082352.zip
          142 kB
        2. collectinfo-2021-04-13T083120-n_0@cb.local.zip
          24.73 MB
        3. collectinfo-2021-04-13T083943-n_0@cb.local.zip
          9.96 MB
        4. log.html
          4.11 MB
        5. output.xml
          24.57 MB
        6. report.html
          245 kB

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-48485 [CBM] Failed to resume backup with error ' (bucket0) (vb 867) Received an unexpected error from the sink callback, beginning teardown'

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Task - A task that needs to be done. MB-46878 Where possible 'cbbackupmgr' should avoid the truncate/overwrite pattern

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Technical task - A technical task. MB-45685 [Backup Service] [Investigate] Database disk image malformed

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            james.lee James Lee added a comment -

            I'm not too worried about this issue at the moment; I've had a chance to look at the code and it looks like 'examinador' is using 'SIGKILL' to terminate the 'cbbackpmgr' process. With this in mind, the following information from the 'write' syscalls man page is interesting.

            write man page

            		 
            If a write() is interrupted by a signal handler before any bytes are written, then the call fails with the error EINTR; if it is interrupted after at least one byte has been written, the call succeeds, and returns the number of bytes written.

            With that in mind, I suspect what's happened here, is that we've killed the backup process whilst it was actually creating an SQLite file; it's quite possible that we've got a partial SQLite file on disk (which would explain the error when attempting query the 'user_version'). Unfortunately we don't have the file so we won't be able to determine for certain so at the moment, this is just a hypothesis.

            If this was the case, I don't believe there's much we could do to rectify this issue since 'SIGKILL' can't be "caught" and handled.

            james.lee James Lee added a comment - I'm not too worried about this issue at the moment; I've had a chance to look at the code and it looks like ' examinador ' is using ' SIGKILL ' to terminate the ' cbbackpmgr ' process. With this in mind, the following information from the ' write ' syscalls man page is interesting. write man page If a write() is interrupted by a signal handler before any bytes are written, then the call fails with the error EINTR; if it is interrupted after at least one byte has been written, the call succeeds, and returns the number of bytes written. With that in mind, I suspect what's happened here, is that we've killed the backup process whilst it was actually creating an SQLite file; it's quite possible that we've got a partial SQLite file on disk (which would explain the error when attempting query the ' user_version '). Unfortunately we don't have the file so we won't be able to determine for certain so at the moment, this is just a hypothesis. If this was the case, I don't believe there's much we could do to rectify this issue since ' SIGKILL ' can't be "caught" and handled.

            People

              james.lee James Lee
              james.lee James Lee
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty