Remove some `Vec` allocations to improve performance

This claws back most of the performance lost in https://github.com/rust-lang/rust/pull/69745.
r? @EddyB