Add forgotten notifier unregister.

Fixes: 99724c18 ("mlxsw: spectrum: Introduce support for router interfaces")
Signed-off-by: NJiri Pirko <jiri@mellanox.com>
Reviewed-by: NIdo Schimmel <idosch@mellanox.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>