Actually, setting HOME to the target user is the behavior of the Debian
sudo config anyway.