All ports types can have a fixed PHY associated with it. Remove the
check which limits removal to only CPU and DSA ports.
Signed-off-by: NAndrew Lunn <andrew@lunn.ch>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>