The intention of this check was to prevent the conflict between
hotplug and removing driver for whatever reason. Currently it
doesn't improve anything and the following rescan process could
still saftly perform the scan flow. So these code seems pointless
now and let's remove them.
Signed-off-by: NShawn Lin <shawn.lin@rock-chips.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>