gpios requested with invalid numbers, or gpios requested from userspace via sysfs
should not try to be deferred on failure.

Cc: stable@kernel.org
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>