Previously, we used the HW GID index in order to search the source GID
in the software GID cached table. In some cases, for example when
the MAC Address of the network interface is changed, the GID cached table
saves the old-IPv6-link-local GID at the end of the table.

When returning the old MAC address, the software GID cached table tries
to add the new IPv6-link-local GID, and when it identifies that the GID
already exists, the software GID cached does not add it. Thus a mismatch
occurs between the HW and the SW GID tables.

It resulted with sending traffic with the wrong source GID.

This commit fixes the issue by taking both from the HW table.

The problem can be reproduced with the following scenario:
Client:
    # ifconfig ens6 2.2.2.5
    # ifconfig ens6 inet6 add 2001:0db8:0:f101::5/64
    # ifconfig ens6 hw ether f4:52:14:61:a0:71
    # ifconfig ens6 inet6 del 2001:0db8:0:f101::5/64
    # ifconfig ens6 inet6 add 2001:0db8:0:f101::5/64
    # ucmatose -f ipv6 -b 2001:0db8:0:f101::5 -s 2001:0db8:0:f101::6 -p 20156
Server:
    # ucmatose -f ipv6 -b 2001:0db8:0:f101::6 -p 20156

Fixes: 4c3eb3ca ('IB/mlx4: Add VLAN support for IBoE')
Signed-off-by: NTalat Batheesh <talatb@mellanox.com>
Reviewed-by: NMaor Gottlieb <maorg@mellanox.com>
Signed-off-by: NLeon Romanovsky <leon@kernel.org>
Signed-off-by: NDoug Ledford <dledford@redhat.com>