Someone had relied on the behavior that preloading with a given scope,
but the behavior has lost in #35496 to fix the minor bug that unloading
through association.

Basically we don't guarantee the internal behavior, but the bugfix can
be achieved without any breaking change, so I've restored the lost
functionality.

Fixes #36638.
Fixes #37720.