The delayed worker isn't needed and is racey. Remove it and do
the state change in line.
Reviewed-by: NYakir Yang <ykk@rock-chips.com>
Tested-by: NYakir Yang <ykk@rock-chips.com>
Signed-off-by: NSean Paul <seanpaul@chromium.org>