GPIO de-bounce clocks don't have any impact on the module idle state, so
the clock code should not wait for the module to enable after the de-bounce
clocks are enabled.

Problem found by Kevin Hilman <khilman@deeprootsystems.com>.
Signed-off-by: NPaul Walmsley <paul@pwsan.com>
Signed-off-by: NKevin Hilman <khilman@deeprootsystems.com>
Signed-off-by: NTony Lindgren <tony@atomide.com>