There is no need to check port for NULL in uart_port_deref() since we call it
only when port is defined.

There are few places that violate this. Fix them here as well.
Signed-off-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>