This ended up causing problems for older parts (particularly ones
using PTEA). Revert this for now, it can be added back in once it's
had some more testing.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>