If we cancel the task_work, the worker will never come into existance.
As this is the last reference to it, ensure that we get it freed
appropriately.

Cc: stable@vger.kernel.org
Reported-by: N진호 <wnwlsgh98@gmail.com>
Signed-off-by: NJens Axboe <axboe@kernel.dk>