When accepting a new IPv4 connect to an IPv6 socket, the CMA tries to
canonize the address family to IPv4, but does not properly process
the listening sockaddr to get the listening port, and does not properly
set the address family of the canonized sockaddr.

Fixes: e51060f0 ("IB: IP address based RDMA connection manager")

Cc: <stable@vger.kernel.org>
Reported-By: NYotam Kenneth <yotamke@mellanox.com>
Signed-off-by: NJason Gunthorpe <jgunthorpe@obsidianresearch.com>
Tested-by: NHaggai Eran <haggaie@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>