This reverts commit f33cf776.

As Johan says, this driver needs a lot more work and these changes are
only going in the wrong direction:
    https://lkml.kernel.org/r/20190523091839.GC568@localhostReported-by: NJohan Hovold <johan@kernel.org>
Cc: Shubhrajyoti Datta <shubhrajyoti.datta@xilinx.com>
Cc: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>