Improve performance around value type structures

The SDK has uses value type structures for the sake of reducing heap allocations, but in some cases they can add other performance penalties.

https://devblogs.microsoft.com/premier-developer/avoiding-struct-and-readonly-reference-performance-pitfalls-with-errorprone-net/

This issue is for a general review to improve the patterns in use.