Clean up the unnecessarily complicated blacklist implementation
somewhat.
Signed-off-by: NJohan Hovold <johan@kernel.org>