We want commands to finish ASAP, so let's remove
that udelay() call.
Signed-off-by: NFelipe Balbi <felipe.balbi@linux.intel.com>