If we're in the process of shutting down the async context, then don't
create new workers if we already have at least the fixed one.
Signed-off-by: NJens Axboe <axboe@kernel.dk>