Initialize structs with {} instead of {0} to make sure that all code is
using the same convention.
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Signed-off-by: NKamal Heib <kamalheib1@gmail.com>
Reviewed-by: NYuval Shaia <yuval.shaia@oracle.com>
Message-Id: <20190314153031.7197-4-kamalheib1@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

The 'rdma_dev_res' parameter is not used in rdma_poll_cq(), so remove it.
Reviewed-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Signed-off-by: NKamal Heib <kamalheib1@gmail.com>
Message-Id: <20190314153031.7197-3-kamalheib1@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

The function's argument rdma_dev_res is not needed as it is stored in
the backend_dev object at init.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Message-Id: <1552300155-25216-9-git-send-email-yuval.shaia@oracle.com>
Reviewed-by: NKamal Heib <kamalheib1@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

When QP is destroyed the backend QP is destroyed as well. This ensures
we clean all received buffer we posted to it.
However, a contexts of these buffers are still remain in the device.
Fix it by maintaining a list of buffer's context and free them when QP
is destroyed.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Message-Id: <1552300155-25216-8-git-send-email-yuval.shaia@oracle.com>
Reviewed-by: NKamal Heib <kamalheib1@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

When device is going down free all saved MAD buffers.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Message-Id: <1552300155-25216-7-git-send-email-yuval.shaia@oracle.com>
Reviewed-by: NKamal Heib <kamalheib1@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

Add counters to enable enhance debugging
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Message-Id: <1552300155-25216-5-git-send-email-yuval.shaia@oracle.com>
Reviewed-by: NKamal Heib <kamalheib1@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

The function rdma_poll_cq is called from two contexts - completion
handler thread which sense new completion on backend channel and
explicitly as result of guest issuing poll_cq command.

Add lock to protect against concurrent executions.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Message-Id: <1552300155-25216-4-git-send-email-yuval.shaia@oracle.com>
Reviewed-by: NKamal Heib <kamalheib1@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

To make code more readable move handling of protected list to a
rdma_utils
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Message-Id: <1552300155-25216-3-git-send-email-yuval.shaia@oracle.com>
Reviewed-by: NKamal Heib <kamalheib1@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

Utilize error_report for all pr_err calls and some pr_dbg that are
considered as errors.
For the remaining pr_dbg calls, the important ones were replaced by
trace points while other deleted.
Some of the functions got renamed to include prefix "rdma/pvrdma"
in the function name.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Message-Id: <1552300155-25216-2-git-send-email-yuval.shaia@oracle.com>
Reviewed-by: NKamal Heib <kamalheib1@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

Do not initialize structs with {0} since some
CLANG versions do not support it.

Use {} construct instead.
Reviewed-by: NYuval Shaia <yuval.shaia@oracle.com>
Tested-by: NCornelia Huck <cohuck@redhat.com>
Message-Id: <20190118124614.24548-3-marcel.apfelbaum@gmail.com>
Reviewed-by: NCornelia Huck <cohuck@redhat.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

To cover the case where fini() was called even when init() fails make
sure objects are not NULL before calling to non-null-safe destructors.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20190116151538.14088-1-yuval.shaia@oracle.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

This member is used only in init_device_caps function, make it local.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: Marcel Apfelbaum<marcel.apfelbaum@gmail.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>

This parameter has no effect, fix it.

The function init_dev_caps sets the front-end's max-sge to MAX_SGE. Then
it checks backend's max-sge and adjust it accordingly (we can't send
more than what the device supports).

On send and recv we need to make sure the num_sge in the WQE does not
exceeds the backend device capability.
This check is done in pvrdma level so check on rdma level is deleted.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Message-Id: <20190109194123.3468-1-yuval.shaia@oracle.com>
Reviewed-by: Marcel Apfelbaum<marcel.apfelbaum@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

With commit 4481985c (rdma: check num_sge does not exceed MAX_SGE)
macro VENDOR_ERR_NO_SGE is no longer in use - delete it.
Signed-off-by: NPrasad J Pandit <pjp@fedoraproject.org>
Reviewed-by: NYuval Shaia <yuval.shaia@oracle.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

rdma back-end has scatter/gather array ibv_sge[MAX_SGE=4] set
to have 4 elements. A guest could send a 'PvrdmaSqWqe' ring element
with 'num_sge' set to > MAX_SGE, which may lead to OOB access issue.
Add check to avoid it.
Reported-by: NSaar Amar <saaramar5@gmail.com>
Signed-off-by: NPrasad J Pandit <pjp@fedoraproject.org>
Reviewed-by: NYuval Shaia <yuval.shaia@oracle.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

Add ability to pass specific WC attributes to CQE such as GRH_BIT flag.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

The control over the RDMA device's GID table is done by updating the
device's Ethernet function addresses.
Usually the first GID entry is determined by the MAC address, the second
by the first IPv6 address and the third by the IPv4 address. Other
entries can be added by adding more IP addresses. The opposite is the
same, i.e. whenever an address is removed, the corresponding GID entry
is removed.

The process is done by the network and RDMA stacks. Whenever an address
is added the ib_core driver is notified and calls the device driver
add_gid function which in turn update the device.

To support this in pvrdma device we need to hook into the create_bind
and destroy_bind HW commands triggered by pvrdma driver in guest.
Whenever a change is made to the pvrdma port's GID table a special QMP
message is sent to be processed by libvirt to update the address of the
backend Ethernet device.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: Marcel Apfelbaum<marcel.apfelbaum@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

MAD (Management Datagram) packets are widely used by various modules
both in kernel and in user space for example the rdma_* API which is
used to create and maintain "connection" layer on top of RDMA uses
several types of MAD packets.

For more information please refer to chapter 13.4 in Volume 1
Architecture Specification, Release 1.1 available here:
https://www.infinibandta.org/ibta-specifications-download/

To support MAD packets the device uses an external utility
(contrib/rdmacm-mux) to relay packets from and to the guest driver.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: Marcel Apfelbaum<marcel.apfelbaum@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

Function create_ah might return NULL, let's exit with an error.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

The field backend_dev->dev is not initialized, fix it.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Message-Id: <20180805153518.2983-14-yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

To ease maintenance of struct comp_thread move all related code to
dedicated function.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Message-Id: <20180805153518.2983-11-yuval.shaia@oracle.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

The structure RdmaRmUserMR has no benefits, remove it an move all its
fields to struct RdmaRmMR.
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Message-Id: <20180805153518.2983-7-yuval.shaia@oracle.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

There are certain operations that are well considered as part of device
configuration while others are needed only when "start" command is
triggered by the guest driver. An example of device initialization step
is msix_init and example of "device start" stage is the creation of a CQ
completion handler thread.

Driver expects such distinction - implement it.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Message-Id: <20180805153518.2983-2-yuval.shaia@oracle.com>
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>

Make sure the backend GID index is less then port's
gid table length.
Signed-off-by: NMarcel Apfelbaum <marcel.apfelbaum@gmail.com>
Reviewed-by: NYuval Shaia <yuval.shaia@oracle.com>
Message-Id: <20180430200223.4119-8-marcel.apfelbaum@gmail.com>

Use the correct printf formats, so that a 32-bit compile doesn't spit
out lots of warnings about %lx being incompatible with uint64_t.
Suggested-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Tested-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20180322095220.9976-4-yuval.shaia@oracle.com>
Signed-off-by: NMarcel Apfelbaum <marcel@redhat.com>

Macro should not cast the given variable to u64 instead it should use
the supplied format argument (fmt).
Reported-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Tested-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20180322095220.9976-3-yuval.shaia@oracle.com>
Signed-off-by: NMarcel Apfelbaum <marcel@redhat.com>

To avoid compilation warnings on 32-bit machines:
rdma_backend.c: In function 'rdma_backend_create_mr':
rdma_backend.c:409:37: error: cast to pointer from integer of different
size [-Werror=int-to-pointer-cast]
	mr->ibmr = ibv_reg_mr(pd->ibpd, (void *)addr, length, access);
Reported-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Tested-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20180322095220.9976-2-yuval.shaia@oracle.com>
Signed-off-by: NMarcel Apfelbaum <marcel@redhat.com>

Our rule right now is to use <> for external headers only.
RDMA code violates that, fix it up.
Signed-off-by: NMichael S. Tsirkin <mst@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>
Reviewed-by: NMarcel Apfelbaum <marcel@redhat.com>
Signed-off-by: NMarcel Apfelbaum <marcel@redhat.com>

This operation is needed by rdma devices - implement it.
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: NMarcel Apfelbaum <marcel@redhat.com>
Signed-off-by: NMarcel Apfelbaum <marcel@redhat.com>

This layer is composed of two sub-modules, backend and resource manager.
Backend sub-module is responsible for all the interaction with IB layers
such as ibverbs and umad (external libraries).
Resource manager is a collection of functions and structures to manage
RDMA resources such as QPs, CQs and MRs.
Reviewed-by: NDotan Barak <dotanb@mellanox.com>
Reviewed-by: NZhu Yanjun <yanjun.zhu@oracle.com>
Signed-off-by: NYuval Shaia <yuval.shaia@oracle.com>
Signed-off-by: NMarcel Apfelbaum <marcel@redhat.com>