This reverts commit 6c7e660a.

The commit causes breakage on several older PXA machines.
Reported-by: NMike Dunn <mikedunn@newsguy.com>
Acked-by: NHaojian Zhuang <haojian.zhuang@linaro.org>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>