The symptom of this issue is that disk snapshoting of MOI index loops indefinitely. From analysis of some customer data - disk snapshot files - it could be seen that the same items are repeatedly being persisted to disk. This can happen when the index item exposed to indexer layer from storage is modified. This modification is not allowed and the exposed index item should be used as read only.
Looking at all the usages of the exposed index items, an issue was identified in the array index mutation processing codepath and was fixed in this commit.
But, some of the clusters where this issue was seen, the issue occurred for a non array index. Here, either there is still some bug in how indexer modifies indexed items or there may be some bug internal to storage in MOI. To isolate the issue further and also provide some extra logging in disk snapshotting codepath, this commit was added and
MB-37375 created. As a blanket fix, this commit introduced an indexer config to expose a safe copy of indexed items to indexer from storage. As such, if the issue is seen again, the indexer config indexer.moi.exposeItemCopy can be set to true as a workaround.
Next Steps - if the issue surfaces again:
- Collect the disk snapshot files for the offending index as described in CBSE-8684
- Check for bad items in the disk snapshot.
- Invesitigate on the line of item being modified illegally.