If the worker isn't on the free_list, don't attempt to delete it.
Signed-off-by: NJens Axboe <axboe@kernel.dk>