Some races with the hardware can happen when we take
ownership of the device. Don't give up after the first try.

Cc: <stable@vger.kernel.org>
Reviewed-by: NJohannes Berg <johannes.berg@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>