No need to allocate and deallocate endpoint memory if the physical link is
down.

Link: https://lore.kernel.org/r/20200408064332.19377-3-mrangankar@marvell.comReviewed-by: NLee Duncan <lduncan@suse.com>
Signed-off-by: NManish Rangankar <mrangankar@marvell.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>