Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25244

MOI: Indefinite disk snapshotting leads to increasing disk usage

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • bug-backlog
    • 4.5.0, 5.0.0
    • storage-engine
    • Untriaged
    • Unknown

    Description

      The symptom of this issue is that disk snapshoting of MOI index loops indefinitely. From analysis of some customer data - disk snapshot files - it could be seen that the same items are repeatedly being persisted to disk. This can happen when the index item exposed to indexer layer from storageĀ  is modified. This modification is not allowed and the exposed index item should be used as read only.

      Looking at all the usages of the exposed index items, an issue was identified in the array index mutation processing codepath and was fixed in this commit.

      But, some of the clusters where this issue was seen, the issue occurred for a non array index. Here, either there is still some bug in how indexer modifies indexed items or there may be some bug internal to storage in MOI. To isolate the issue further and also provide some extra logging in disk snapshotting codepath, this commit was added and MB-37375 created. As a blanket fix, this commit introduced an indexer config to expose a safe copy of indexed items to indexer from storage. As such, if the issue is seen again, the indexer config indexer.moi.exposeItemCopy can be set to true as a workaround.

      Next Steps - if the issue surfaces again:

      1. Collect the disk snapshot files for the offending index as described in CBSE-8684
      2. Check for bad items in the disk snapshot.
      3. Invesitigate on the line of item being modified illegally.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              akhil.mundroy Akhil Mundroy
              sarath Sarath Lakshman
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty