Improve ForestDB detection of corrupted indexes and provide a mechanism to alert users to the need for rebuilding them

When ForestDB indexes are corrupted, it is possible for the Index service to continue operating, however there are some side effects and consequences that are not readily apparent.

For example, when upgrading from CB 4.0 or CB 4.1 to CB 4.5+, there is a forced compaction process after the upgrade in order to upgrade the ForestDB file format (see MB-18478). If the ForestDB index files are corrupt before this compaction occurs, then the upgrade process will continuously attempt to compact the index files, fail to do so, and retry again forever. The only workaround is to be aware that the indexes are corrupt and need to be dropped and recreated. Unfortunately, there is no way for end users to know this and the only way to detect for corrupt index files would be to scan the indexer.log file for FDB ERR messages that indicate corruption.

In addition, if the Index service is started and encounters corrupt index files, it will need to do a complete of the scan from the end of the file to the beginning. During this time the Index service stays in bootstrap mode and cannot service indexing requests. Since customers are not alerted to the fact that the indexes are corrupt, they can wait days for the Index service to finish scanning the corrupt index files before moving from bootstrap to a ready state. During this time no indexing is occurring, which means an accumulation of indexing backlog as well as production queries being returned stale result sets.

There should be a mechanism to detect for corruption and to bring it to end user attention through the cluster admin UI as well as through cluster notifications/alerts. This will allow end users to be aware of the issue and understand that they need to rebuild their indexes to avoid inconvenient side effects such as the ones mentioned above.