This reverts commit 70f3a9ca.

That commit was causing a lockdep splat with g_ether and that
was interfering with proper functionality.
Signed-off-by: NFelipe Balbi <balbi@ti.com>