This breaks creating unaligned raw pointers via `&packed.field as *const _`, which needs to be legal.
Also it doesn't seem like LLVM still relies on this, see
* https://github.com/solson/miri/issues/244#issuecomment-315563640
* https://internals.rust-lang.org/t/rules-for-alignment-and-non-nullness-of-references/5430/16

We probably want to handle this invariant like the others that validation is concerned with, and only
check it on function boundaries for now.