Pod Recovery Must Be Based on Original Image

When we test killing pods during an upgrade with a persistent storage backed cluster, we only test killing the new pod, and not the old pod.  If we were to do so, then the old pod would be recovered with a new image, as we cannot know what the old image was by that point.  The result is that the new couchbase is incompatible with the old couchbase's data structures, and cannot perform an online migration as that is done in post-install scripts in the package manager – containers should be immutable so using a package manager to do the upgrade is out of the question.  While in theory you should be able to rollback the upgrade, this doesn't appear possible from my simple manual testing, and thus needs more investigation.

We need to provide a PVC annotation with the EXACT image that was used to create this data, then use that for recovery.  This will need the tests extending, and an upgrade path to generate annotations on existing pods, tests etc.