During IB device removal, cancel the event worker before the device
structure is freed.

Fixes: a4cf0443 ("smc: introduce SMC as an IB-client")
Reported-by: syzbot+b297c6825752e7a07272@syzkaller.appspotmail.com
Signed-off-by: NKarsten Graul <kgraul@linux.ibm.com>
Reviewed-by: NUrsula Braun <ubraun@linux.ibm.com>
Reviewed-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>