Clean up the exit path during .remove() and .shutdown() calls when in
Endpoint mode.

Link: https://lore.kernel.org/r/20220721142052.25971-14-vidyas@nvidia.comSigned-off-by: NVidya Sagar <vidyas@nvidia.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>