While debugging OS crashes due to firmware crashes, I enabled
kasan, and it noticed that peer objects were being used-after-freed.

Looks like there are two places we could be leaving stale references
in the peer-map, so clean that up.
Signed-off-by: NBen Greear <greearb@candelatech.com>
Signed-off-by: NKalle Valo <kvalo@qca.qualcomm.com>