Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-22718

Fix file cleanup logic in FDB GSI compaction

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • bug-backlog
    • 4.1.2, 4.5.1, 4.6.0, 5.0.0
    • forestdb
    • None
    • Untriaged
    • No

    Description

      Now tracking:
      We need a call in forestdb that takes the base file (say data.fdb) and scans available files (data.fdb.0 data.fdb.1, data.fdb.2 etc) and returns the correct file to open, and files which can be safely deleted. This would not only take care of below, but also issue seen in CBSE-3797, and hence I'd like to request this for current release to fix file open and unwanted file truncate logic in a robust manner.

      Was originally:
      Indexer has a cleanup mechanism to deal with forestdb compaction failures. This can lead to issue if Indexer is holding on to a snapshot from the old file after compaction finishes.

      e.g.

      1. A compaction run starts when mutations are being processed. The mutation load finishes during the duration of the run. Indexer creates a snapshot and keeps its open to be able to process query requests faster. This snapshot is being referenced from the old file.

      2. Forestdb finishes compaction and new file is ready. However it cannot let go of the old file as Indexer is keeping the snapshot open. This leads to 2 files being present on the disk.

       

      default_primary1_11043795554325603859_0.index > ll
      total 4397992
      rw-rr- 1 deepkaran staff 863502336 Feb 7 17:05 data.fdb.1
      rw-rr- 1 deepkaran staff 1388269568 Feb 7 17:05 data.fdb.0

       

      3. If there is any mutation after this, Indexer will let go of the old snapshot and this will cause forestdb to delete the old file. [This is the normal outcome].

      4. If there are no mutations before the next compaction check and If the last compaction run couldn't bring down the fragmentation below the "min_frag" percentage(which means compaction needs to be triggered again), this issue will surface.

      5. Indexer will start a compaction run and find that there are 2 files on disk. This is treated as failure of the last compaction and the latest file is removed. This leads to next compaction run failing with error "invalid args".

      Attachments

        For Gerrit Dashboard: MB-22718
        # Subject Branch Project Status CR V

        Activity

          People

            prathibha Prathibha Bisarahalli (Inactive)
            deepkaran.salooja Deepkaran Salooja
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:

              PagerDuty