If we stop the kthread, we may end up up'ing the sem twice, which seems
unintended.
Reported-by: NDan Carpenter <error27@gmail.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>