Otherwise the reference count goes negative as IPv6 packets complete.

Fixes: 4ed6ad1e ("IB/rxe: Cache dst in QP instead of getting it...")
Signed-off-by: NAndrew Boyer <andrew.boyer@dell.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>