Perform cleanup also in non-master flow.
The VFs use communication channel as well.
Signed-off-by: NEugenia Emantayev <eugenia@mellanox.co.il>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>