If port.line is out of range, we still need to release some resources, or
we will leak them.

Fixes: afc7851f ("serial: pxa: Fix out-of-bounds access through serial port index")
Signed-off-by: NChristophe JAILLET <christophe.jaillet@wanadoo.fr>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>