RQ WQE size other than 128B is not supported.  Correct
RQ size calculation to use 128B only.

Since this breaks ABI, add additional code to
provide compatibility with v4 user provider, libi40iw.
Signed-off-by: NChien Tin Tung <chien.tin.tung@intel.com>
Signed-off-by: NHenry Orosco <henry.orosco@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Add struct ib_udata to the signature of create_ah callback that is
implemented by IB device drivers. This allows HW drivers to return extra
data to the userspace library.
This patch prepares the ground for mlx5 driver to resolve destination
mac address for a given GID and return it to userspace.
This patch was previously submitted by Knut Omang as a part of the
patch set to support Oracle's Infiniband HCA (SIF).
Signed-off-by: NKnut Omang <knut.omang@oracle.com>
Signed-off-by: NMoni Shoua <monis@mellanox.com>
Reviewed-by: NYishai Hadas <yishaih@mellanox.com>
Signed-off-by: NLeon Romanovsky <leon@kernel.org>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Some resources are incorrectly organized and at odds with
HW capabilities. Specifically, ILQ, IEQ, QPs, MSS, QOS
and statistics belong in a VSI.
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NHenry Orosco <henry.orosco@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Add a QP reference when terminate timer is started to ensure
the destroy QP doesn't race ahead to free the QP while it is being
referenced in the terminate timer's handler.
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

hw_stats is a pointer to i40_iw_dev_stats struct in i40iw_get_hw_stats().
Use hw_stats and not &hw_stats in the memcpy to copy the i40iw device stats
data into rdma_hw_stats counters.

Fixes: b40f4757 ("IB/core: Make device counter infrastructure dynamic")

Cc: stable@vger.kernel.org # 4.7+
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

On i40iw device close, disconnect all connected QPs by moving
them to error state; and block further QPs, PDs and CQs from
being created. Additionally, make sure all resources have been
freed before deallocating the ibdev as part of the device close.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Add support to allow each independent memory region to
be configured for 2MB page size in addition to 4KB
page size.
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NHenry Orosco <henry.orosco@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Add support to use physically mapped WQ's and MR's if determined
that the OS registered user-memory for the region is physically
contiguous. This feature will eliminate the need for unnecessarily
setting up and using PBL's when not required.
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NHenry Orosco <henry.orosco@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

In i40iw_post_send, use the actual page size instead of
encoded page size. This is to be consistent with the
rest of the file.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NHenry Orosco <henry.orosco@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Some resources are consumed internally and not available to the user.
After hw is initialized, figure out how many resources are consumed
and subtract those numbers from the initial max device capability in
i40iw_query_device().
Signed-off-by: NHenry Orosco <henry.orosco@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

When creating QPs, ensure init_attr->cap.max_recv_sge
is clipped to MAX_FRAG_COUNT.

Expose MAX_FRAG_COUNT for max_recv_sge and max_send_sge in
i40iw_query_qp().
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NHenry Orosco <henry.orosco@intel.com>
Reviewed-By: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Assign each CEQ vector to a different CPU when possible, then
when creating a CQ, use the vector for the CEQ id. This
allows completion work to be distributed over multiple cores.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NHenry Orosco <henry.orosco@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Add support for QoS on QPs. Upon device initialization,
a map is created from user priority to queue set
handles. On QP creation, use ToS to look up the queue
set handle for use with the QP.
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NHenry Orosco <henry.orosco@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

The prints after [k|v][m|z|c]alloc() functions are not needed,
because in case of failure, allocator will print their internal
error prints anyway.
Signed-off-by: NLeon Romanovsky <leon@kernel.org>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

iwpbl->iwmr points to the structure that contains iwpbl,
which is iwmr. Setting this to NULL would result in
writing to freed memory. So just free iwmr, and return.

Fixes: d3749841 ("i40iw: add files for iwarp interface")
Reported-by: NStefan Assmann <sassmann@redhat.com>
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Memory allocated for iwqp; iwqp->allocated_buffer is freed twice in
the create_qp error path. Correct this by having it freed only once in
i40iw_free_qp_resources().

Fixes: d3749841 ("i40iw: add files for iwarp interface")
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Post_cq parameter passed to i40iw_cq_poll_completion is always
true; so remove.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Fix size parameter passed to i40iw_reg_phys_mr and use it to
register memory.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Fix incorrect usage of ENOSYS and other return codes.
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Fix to enable remote access rights when allocating stag.

Fixes: b7aee855 ("RDMA/i40iw: Add base memory management extensions")
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

And remove sysfs support in favor of the core version.
Reviewed-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NIra Weiny <ira.weiny@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Set the chunk_size to enable level-1 PBL support when the fast memory
page count is more than one.
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Return correct value for max_fast_reg_page_list_len from
i40iw_query_device().
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

i40iw_get_pble returns 0 on success. Correct the check on return
code.
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

CQ is armed for solicited events only, ignoring other notification
flags. Correct this by arming for next and arming for solicited
event if IB_CQ_SOLICITED is set. Also protect CQ shadow area update
with spinlock.
Signed-off-by: NShiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

In practice, each RDMA device has a unique set of counters that the
hardware implements.  Having a central set of counters that they must
all adhere to is limiting and causes many useful counters to not be
available.

Therefore we create a dynamic counter registration infrastructure.

The driver must implement a stats structure allocation routine, in
which the driver must place the directory name it wants, a list of
names for all of the counters, an array of u64 counters themselves,
plus a few generic configuration options.

We then implement a core routine to create a sysfs file for each
of the named stats elements, and a core routine to retrieve the
stats when any of the sysfs attribute files are read.

To avoid excessive beating on the stats generation routine in the
drivers, the core code also caches the stats for a short period of
time so that someone attempting to read all of the stats in a
given device's directory will not result in a stats generation
call per file read.

Future work will attempt to standardize just the shared stats
elements, and possibly add a method to get the stats via netlink
in addition to sysfs.
Signed-off-by: NChristoph Lameter <cl@linux.com>
Signed-off-by: NMark Bloch <markb@mellanox.com>
Reviewed-by: NSteve Wise <swise@opengridcomputing.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>
[ Add caching, make structure names more informative, add i40iw support,
  other significant rewrites from the original patch ]

The SRP initiator allows to set max_sectors to a value that exceeds
the largest amount of data that can be mapped at once with an mlx4
HCA using fast registration and a page size of 4 KB. Hence modify
ib_map_mr_sg() such that it can map partial sg-elements. If an
sg-element has been mapped partially, let the caller know
which fraction has been mapped by adjusting *sg_offset.
Signed-off-by: NBart Van Assche <bart.vanassche@sandisk.com>
Tested-by: NLaurence Oberman <loberman@redhat.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Signed-off-by: NChristoph Hellwig <hch@lst.de>
Tested-by: NSteve Wise <swise@opengridcomputing.com>
Reviewed-by: NBart Van Assche <bart.vanassche@sandisk.com>
Reviewed-by: NSagi Grimberg <sagi@grimberg.me>
Reviewed-by: NSteve Wise <swise@opengridcomputing.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Fix for checking if the QP associated with a completion
has been destroyed while processing CQ elements.
If that is the case, move the CQ head to the next element
and continue completion processing.
Signed-off-by: NTatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

A check is added to validate the requested sge number.
iWARP doesn't support multiple sg elements for
RDMA READ work requests.
Signed-off-by: NTatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Fix to calculate the SQ size based on the max
frag_count, requested by the application instead
of overwriting it with the max supported frag_count
Signed-off-by: NTatyana Nikolova <tatyana.e.nikolova@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Adding sq and rq drain functions, which block until all
previously posted wr-s in the specified queue have completed.
A completion object is signaled to unblock the thread,
when the last cqe for the corresponding queue is processed.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Implement fast register mr, Local invalidate, send with
invalidate and RDMA read with invalidate.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Queue users of virtual channel on a waitqueue until the channel is
clear instead of failing the call when the channel is occupied.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Populate PCI info fields from PCI device structure.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Add a check for cq_poll_info.error before setting vendor_err
instead of always setting it.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

iwqp->allocated_buffer is a self-referencing pointer to iwqp.
Do not set iwqp->allocated_buffer to NULL after freeing it.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Fix to correct max reported message size in query port.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NFaisal Latif <faisal.latif@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Include inline data size as part of SQ size calculation.
RQ size calculation uses only number of SGEs and does not
support 96 byte WQE size.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Change region_length to u64 as a region can be > 4GB.
Signed-off-by: NMustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>