Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44215

[BP] Validity bit is used to determine corrupted merged components

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • CX Sprint 236

    Description

      We currently maintain the maximum sequence number of valid components (btrees) of an index in the index checkpoint file. The checkpoint file is updated after all of the btree files have been forced to disk. On node start up, if any btree found with a sequence number > the one on the checkpoint file, that btree is deleted. However, since a merged component have the same high sequence as the btrees that are merged, we cannot determine using the same mechanism whether this merged component was successfully written to disk or not. For that reason, we currently use a validity bit on the btree metedata page to check the validity of such component.

      When writing a merged component to disk with compression enabled and some unexpected failure happens at a point where the btree file (#_#_b) and the btree compressed pages dictionary file (#_#_b.dic) are out of sync, then there is possibility that we might fail to read the metadata page of this component which will result in the node failing to startup.

      A safe workaround to get out of this startup failure is to manually delete a merged component whose original components still exists on disk. For example, if we are attempting to merge components with sequence from 0 to 2, then the following files would be on disk:

      0_0_b
      0_0_b.dic
      0_0_f
       
      1_1_b
      1_1_b.dic
      1_1_f
       
      2_2_b
      2_2_b.dic
      2_2_f
       
      0_2_b
      0_2_b.dic
      0_2_f
      

      In this case, it is safe to delete the files of the merged component (0_2) since all the original btrees with sequences 0 to 2 still exist on disk. No rollback will be necessary when this workaround is performed.

      Attachments

        Issue Links

          Activity

            People

              murtadha.hubail Murtadha Hubail
              murtadha.hubail Murtadha Hubail
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty