The call to sdma_progress() is called outside the wait lock.

In this case, there is a race condition where sdma_progress() can return
false and the sdma_engine can idle.  If that happens, there will be no
more sdma interrupts to cause the wakeup and the vnic_sdma xmit will hang.

Fix by moving the lock to enclose the sdma_progress() call.

Also, delete the tx_retry. The need for this was removed by:
commit bcad2913 ("IB/hfi1: Serve the most starved iowait entry first")

Fixes: 64551ede ("IB/hfi1: VNIC SDMA support")
Reviewed-by: NGary Leshner <Gary.S.Leshner@intel.com>
Signed-off-by: NMike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

This patch adds an interface to allow the driver to initialize the QP priv
struct when the QP is created and after the qpn has been assigned.  A
field is added to the QP priv struct to reference the rcd and two new
files are added to contain the function to initialize the rcd field so
that more TID RDMA related code can be added here later.
Signed-off-by: NMike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: NKaike Wan <kaike.wan@intel.com>
Signed-off-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

The OPFN and TID RDMA capability bits are added to allow users to control
which feature is enabled and disabled.
Reviewed-by: NMike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: NMitko Haralanov <mitko.haralanov@intel.com>
Signed-off-by: NKaike Wan <kaike.wan@intel.com>
Signed-off-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

Currently, When a reserved operation is completed, its entry in the send
queue will not be unreserved, which leads to the miscalculation of
qp->s_avail and thus the triggering of a WARN_ON call trace. This patch
fixes the problem by unreserving the reserved operation when it is
completed.

Fixes: 856cc4c2 ("IB/hfi1: Add the capability for reserved operations")
Reviewed-by: NMike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: NKaike Wan <kaike.wan@intel.com>
Signed-off-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

Ingress packet check for 16B/bypass packets should consider the port
LMC. Not doing this will result in packets sent to the LMC LIDs getting
dropped. The check is implemented in HW for 9B packets.
Reviewed-by: NMike Ruhl <michael.j.ruhl@intel.com>
Signed-off-by: NAshutosh Dixit <ashutosh.dixit@intel.com>
Signed-off-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

An incorrect sge sizing in the HFI PIO path will cause an OOPs similar to
this:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [] hfi1_verbs_send_pio+0x3d8/0x530 [hfi1]
PGD 0
Oops: 0000 1 SMP
 Call Trace:
 ? hfi1_verbs_send_dma+0xad0/0xad0 [hfi1]
 hfi1_verbs_send+0xdf/0x250 [hfi1]
 ? make_rc_ack+0xa80/0xa80 [hfi1]
 hfi1_do_send+0x192/0x430 [hfi1]
 hfi1_do_send_from_rvt+0x10/0x20 [hfi1]
 rvt_post_send+0x369/0x820 [rdmavt]
 ib_uverbs_post_send+0x317/0x570 [ib_uverbs]
 ib_uverbs_write+0x26f/0x420 [ib_uverbs]
 ? security_file_permission+0x21/0xa0
 vfs_write+0xbd/0x1e0
 ? mntput+0x24/0x40
 SyS_write+0x7f/0xe0
 system_call_fastpath+0x16/0x1b

Fix by adding the missing sizing check to correctly determine the sge
length.

Fixes: 77241056 ("IB/hfi1: add driver files")
Reviewed-by: NMike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: NMichael J. Ruhl <michael.j.ruhl@intel.com>
Signed-off-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

VNIC assumes that all SDMA engines have been configured for use.  This is
not necessarily true (i.e. if the count was constrained by the module
parameter).

Update VNICs usage to use the configured count, rather than the hardware
count.
Reviewed-by: NMike Marciniszyn <mike.marciniszyn@intel.com>
Reviewed-by: NGary Leshner <gary.s.leshner@intel.com>
Signed-off-by: NMichael J. Ruhl <michael.j.ruhl@intel.com>
Signed-off-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

A CA is supposed to ignore FECN bits in multicast, ACK, and CNP
packets. This patch corrects the behavior of the HFI1 driver in this
regard by ignoring FECNs in those packet types.

While fixing the above behavior, fix the extraction of the FECN and BECN
bits from the packet headers for both 9B and 16B packets.

Furthermore, this patch corrects the driver's response to a FECN in RDMA
READ RESPONSE packets. Instead of sending an "empty" ACK, the driver now
sends a CNP packet. While editing that code path, add the missing trace
for CNP packets.

Fixes: 88733e3b ("IB/hfi1: Add 16B UD support")
Fixes: f59fb9e0 ("IB/hfi1: Fix handling of FECN marked multicast packet")
Reviewed-by: NKaike Wan <kaike.wan@intel.com>
Reviewed-by: NMike Marciniszyn <mike.marciniszyn@intel.com>
Reviewed-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NMitko Haralanov <mitko.haralanov@intel.com>
Signed-off-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

When it is requested to change its physical state back to Offline while in
the process to go up, DC8051 will set the ERROR field in the
DC8051_DBG_ERR_INFO_SET_BY_8051 register. This ERROR field will remain
until the next time when DC8051 transitions from Offline to Polling.
Subsequently, when the host requests DC8051 to change its physical state
to Polling again, it may receive a DC8051 interrupt with the stale ERROR
field still in DC8051_DBG_ERR_INFO_SET_BY_8051. If the host link state has
been changed to Polling, this stale ERROR will force the host to
transition to Offline state, resulting in a vicious cycle of Polling
->Offline->Polling->Offline. On the other hand, if the host link state is
still Offline when the stale ERROR is received, the stale ERROR will be
ignored, and the link will come up correctly.  This patch implements the
correct behavior by changing host link state to Polling only after DC8051
changes its physical state to Polling.
Reviewed-by: NMike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: NKrzysztof Goreczny <krzysztof.goreczny@intel.com>
Signed-off-by: NKaike Wan <kaike.wan@intel.com>
Signed-off-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

This patch dumps the pio info for non-user send contexts to assist
debugging in the field.
Reviewed-by: NMike Marciniczyn <mike.marciniszyn@intel.com>
Reviewed-by: NMike Ruhl <michael.j.ruhl@intel.com>
Signed-off-by: NKaike Wan <kaike.wan@intel.com>
Signed-off-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

This patch implements the process flow of SRQ asynchronous
event.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

This patch implements the SRQ(Share Receive Queue) verbs
and update the poll cq verbs to deal with SRQ complentions.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

This patch inits hem resource for SRQ table, includes
SRQWQE and SRQWQE index resource.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

This patch configures the flags for enabling the
SRQ(Share Receive Queue) capacity as well as update the
verb of querying device for setting srq specifications.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

From Yishai,
-----------------------------------
This series enriches DEVX support in few aspects: it enables interoperability
between DEVX and verbs and improves mechanism for controlling privileged DEVX
commands.

The first patch updates mlx5 ifc file.

Next 3 patches enable modifying and querying verbs objects via the DEVX
interface.

To achieve that the core layer introduced the 'UVERBS_IDR_ANY_OBJECT' type
to match any IDR object. Once it's used by some driver's method, the
infrastructure skips checking for the IDR type and it becomes the driver
handler responsibility.

The DEVX methods of modify and query were changed to get any object type via
the 'UVERBS_IDR_ANY_OBJECT' mechanism. The type checking is done per object as
part of the driver code.

The next 3 patches introduce more robust mechanism for controlling privileged
DEVX commands. The responsibility to block/allow per command was moved to be
done in the firmware based on the UID credentials that the driver reports upon
user context creation. This enables more granularity per command based on the
device security model and the user credentials.

In addition, by introducing a valid range for 'general commands' we prevent the
need to touch the driver's code any time that a new future command will be
added.

The last patch fixes the XRC verbs flow once a DEVX context is used. This is
needed as XRCD is some shared kernel resource and as such a kernel UID (=0)
should be used in its related resources.

Thanks

Yishai Hadas
-----------------------------------

The top 6 patches are the mlx5-devx series, the remainder are from the
mlx5-next tree as the mlx5-devx series depended on the mlx5-next
mlx5_ifc file update.

* mlx5-devx: (42 commits)
  IB/mlx5: Allow XRC usage via verbs in DEVX context
  IB/mlx5: Update the supported DEVX commands
  IB/mlx5: Enforce DEVX privilege by firmware
  IB/mlx5: Enable modify and query verbs objects via DEVX
  IB/core: Enable getting an object type from a given uobject
  IB/core: Introduce UVERBS_IDR_ANY_OBJECT
  net/mlx5: Update mlx5_ifc with DEVX UCTX capabilities bits
  RDMA/mlx5: Unfold modify RMP function
  RDMA/mlx5: Unfold create RMP function
  RDMA/mlx5: Initialize SRQ tables on mlx5_ib
  RDMA/mlx5: Update SRQ functions signatures to mlx5_ib format
  RDMA/mlx5: Use stages for callback to setup and release DEVX
  RDMA/mlx5: Remove SRQ signature global flag
  net/mlx5: Move SRQ functions to RDMA part
  net/mlx5: Remove references to local mlx5_core functions
  net/mlx5: Remove not-used lib/eq.h header file
  net/mlx5: Remove dead transobj code
  net/mlx5: Align SRQ licenses and copyright information
  net/mlx5: Debug print for forwarded async events
  net/mlx5: Forward SRQ resource events
  ...
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Allows XRC usage from the verbs flow in a DEVX context.
As XRCD is some shared kernel resource between processes it should be
created with UID=0 to point on that.

As a result once XRC QP/SRQ are created they must be used as well with
UID=0 so that firmware will allow the XRCD usage.
Signed-off-by: NYishai Hadas <yishaih@mellanox.com>
Reviewed-by: NArtemy Kovalyov <artemyko@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Update the supported DEVX commands, it includes adding to the
query/modify command's list and to the encoding handling.

In addition, a valid range for general commands was added to be used for
future commands.
Signed-off-by: NYishai Hadas <yishaih@mellanox.com>
Reviewed-by: NArtemy Kovalyov <artemyko@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Enforce DEVX privilege by firmware, this enables future device
functionality without the need to make driver changes unless a new
privilege type will be introduced.
Signed-off-by: NYishai Hadas <yishaih@mellanox.com>
Reviewed-by: NArtemy Kovalyov <artemyko@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Enables modify and query verbs objects via the DEVX interface.
To support this the above DEVX handlers were changed to get any
object type via the UVERBS_IDR_ANY_OBJECT mechanism.

The type checking and handling is done per object as part of the
driver code.
Signed-off-by: NYishai Hadas <yishaih@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Enable getting an object type from a given uobject, the type is saved
upon tree merging and is returned as part of some helper function.
Signed-off-by: NYishai Hadas <yishaih@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Introduce the UVERBS_IDR_ANY_OBJECT type to match any IDR object.

Once used, the infrastructure skips checking for the IDR type, it
becomes the driver handler responsibility.

This enables drivers to get in a given method an object from various of
types.
Signed-off-by: NYishai Hadas <yishaih@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

The enhanced devx support series needs commit:
9d43faac ("net/mlx5: Update mlx5_ifc with DEVX UCTX capabilities bits")
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Expose device capabilities for DEVX user context, it includes which caps
the device is supported and a matching bit to set as part of user
context creation.
Signed-off-by: NYishai Hadas <yishaih@mellanox.com>
Reviewed-by: NArtemy Kovalyov <artemyko@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

There is no need to perform modify_rmp in two separate function,
while one of them uses stack as a placeholder for data while other
allocates it dynamically. Combine those two functions to one call
instead of two.
Reviewed-by: NMark Bloch <markb@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

There is no need to perform create_rmp in two separate function, while
one of them uses stack as a placeholder for data while other allocates
it dynamically. Combine those two functions to one instead of two.
Reviewed-by: NMark Bloch <markb@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

Transfer initialization and cleanup from mlx5_priv struct of
mlx5_core_dev to be part of mlx5_ib_dev. This completes removal
of SRQ from mlx5_core.
Reviewed-by: NMark Bloch <markb@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

Reflect the change of moving SRQ code from mlx5_core to mlx5_ib by
updating function signatures do not require mlx5_core_dev as an input,
because all operations in mlx5_ib are supposed to use mlx5_ib_dev.
Reviewed-by: NMark Bloch <markb@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

Reuse existing infrastructure to initialize and release DEVX uid.
The DevX interface is intended for user space access, so it is supposed
to be initialized before ib_register_device(). Also it isn't supported
in switchdev mode and don't need to initialize it in that mode.

Fixes: 76dc5a84 ("IB/mlx5: Manage device uid for DEVX white list commands")
Reviewed-by: NMark Bloch <markb@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

SRQ signature is not supported, hence no need for special static
global variable to announce it.
Reviewed-by: NMark Bloch <markb@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

There is no need to keep SRQ which is RDMA object in mlx5_core.
In this patch, we partially move the execution code, while next patches
will move table initialization/release logic too.
Reviewed-by: NMark Bloch <markb@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

As a preparation to move SRQ functionality to RDMA, drop all references
to mlx5_core logic and make SRQ be dependent on shared code only.

Most of the time, we are interested to know if events are working/not
working and it is possible with previous commit ("net/mlx5: Debug print
for forwarded async events").
Reviewed-by: NMark Bloch <markb@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

lib/eq.h is needed for EQ manipulation which are not performed in SRQ.
Reviewed-by: NMark Bloch <markb@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

Delete functions which are not called and not needed.
Reviewed-by: NMark Bloch <markb@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

Ensure that both RDMA and netdev parts of SRQ implementation
has same copyright and license information annotated by SPDX
tags.
Reviewed-by: NMark Bloch <markb@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>

[leonro@server ~]$ rdma res show
1: mlx5_0: pd 3 cq 5 qp 4 cm_id 0 mr 0 ctx 0
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Add restrack annotations to track allocations of ucontexts.
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Add ability to track allocated ib_ucontext, which are limited
resource and worth to be visible by users.
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Make all of the write() handlers use a consistent flow

From Jason,

This series unifies all the write handlers to use a flow that is very
similar to the ioctl handler flow, including having the same basic
assumptions about extensible buffer handling and the same handler
function call signature.

Along the way this consolidates all the copy_to/from_user into a small
set of safe buffer accessor functions tailored to the usage here. These
accessors use the new dispatcher-controlled calling convention for ucore
data, and support a placement of the response that does not rely on the
cmd.response value.

Overall this brings in in strong bounds checking to all the write()
handlers and consistent enforcement of the zero-fill/zero-check
methodology for buffer extension.

The end result is a significant complexity reduction for all of the
handlers and creates a high degree of uniformity between the write,
write_ex, and ioctl handlers and dispatch flow.

Thanks

Jason Gunthorpe (12):
  RDMA/uverbs: Remove out_len checks that are now done by the core
  RDMA/uverbs: Use uverbs_attr_bundle to pass ucore for write/write_ex
  RDMA/uverbs: Get rid of the 'callback' scheme in the compat path
  RDMA/uverbs: Use uverbs_response() for remaining response copying
  RDMA/uverbs: Use uverbs_request() for request copying
  RDMA/uverbs: Use uverbs_request() and core for write_ex handlers
  RDMA/uverbs: Fill in the response for IB_USER_VERBS_EX_CMD_MODIFY_QP
  RDMA/uverbs: Simplify ib_uverbs_ex_query_device
  RDMA/uverbs: Add a simple iterator interface for reading the command
  RDMA/uverbs: Use the iterator for ib_uverbs_unmarshall_recv()
  RDMA/uverbs: Do not check the input length on create_cq/qp paths
  RDMA/uverbs: Use only attrs for the write() handler signature

 drivers/infiniband/core/rdma_core.h   |    5 +-
 drivers/infiniband/core/uverbs_cmd.c  | 1165 ++++++++++---------------
 drivers/infiniband/core/uverbs_main.c |   23 +-
 drivers/infiniband/core/uverbs_uapi.c |   23 +-
 include/rdma/uverbs_ioctl.h           |    9 +-
 5 files changed, 479 insertions(+), 746 deletions(-)
Signed-off-by: NDoug Ledford <dledford@redhat.com>

All of the old arguments can be derived from the uverbs_attr_bundle
structure, so get rid of the redundant arguments. Most of the prior work
has been removing users of the arguments to allow this to be a simple
patch.
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

If the user did not provide a long enough command buffer then the missing
bytes are forced to zero. There is no reason to check the length if a zero
value is OK.
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>
Signed-off-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>