mlx4_table_find (for FMR MPTs) requires that ICM memory already be
mapped.  Before this fix, FMR allocation depended on ICM memory
already being mapped for the MPT entry.  If all currently mapped
entries are taken, the find operation fails (even if the MPT ICM table
still had more entries, which were just not mapped yet).

This fix moves the mpt find operation to fmr_enable, to guarantee that
any required ICM memory mapping has already occurred.

Found by Oren Duer of Mellanox.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Commit 313abe55 ("mlx4_core: For 64-bit systems, vmap() kernel queue
buffers") caused this to pop up on powerpc allyesconfig, looks like a
missing include file:

    drivers/net/mlx4/alloc.c: In function 'mlx4_buf_alloc':
    drivers/net/mlx4/alloc.c:162: error: implicit declaration of function 'vmap'
    drivers/net/mlx4/alloc.c:162: error: 'VM_MAP' undeclared (first use in this function)
    drivers/net/mlx4/alloc.c:162: error: (Each undeclared identifier is reported only once
    drivers/net/mlx4/alloc.c:162: error: for each function it appears in.)
    drivers/net/mlx4/alloc.c:162: warning: assignment makes pointer from integer without a cast
    drivers/net/mlx4/alloc.c: In function 'mlx4_buf_free':
    drivers/net/mlx4/alloc.c:187: error: implicit declaration of function 'vunmap'
Signed-off-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Now that struct mlx4_buf.u is a struct instead of a union because of
the vmap() changes, there's no point in having a struct at all.  So
move .direct and .page_list directly into struct mlx4_buf and get rid
of a bunch of unnecessary ".u"s.
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Since kernel virtual memory is not a problem on 64-bit systems, there
is no reason to use our own 2-layer page mapping scheme for large
kernel queue buffers on such systems.  Instead, map the page list to a
single virtually contiguous buffer with vmap(), so that can we access
buffer memory via direct indexing.
Signed-off-by: NMichael S. Tsirkin <mst@dev.mellanox.co.il>
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Signed-off-by: NRoland Dreier <rolandd@cisco.com>

The firmware QUERY_ADAPTER command does not return vendor_id,
device_id, and revision_id; eliminate these fields from the query.

Initialize the rev_id field of the mlx4 device via init_node_data (MAD
IFC query), as is done in the query_device verb implementation.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

    
Commit 3d73c288 ("mlx4_core: Fix section mismatches") fixed some of
the section mismatches introduced when error recovery was added, but
there were still more cases of errory recovery code calling into
__devinit code from regular .text.  Fix this by getting rid of the
now-incorrect __devinit annotations.
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

log_max_eqs is a 4-bit field, not a 3-bit field in the response to the
QUERY_DEV_CAP FW command, so we should mask with 0xf instead of 0x7
when reading it.

Found by Yossi Leybovitch of Mellanox.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

When checking the states passed in, mlx4_qp_modify() accidentally checks
cur_state twice rather than checking cur_state and new_state.  Fix this
to make sure that both values are in-bounds.

Since these values may be passed in from userspace, this bug results in
userspace being able to trigger an oops.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Cc: stable <stable@kernel.org>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Fix thinko in commit eaf559bf ("mlx4_core: Don't free special QPs in
QP number bitmap").  The old commit had the logic exactly backwards
and ended up freeing *only* special QPs, which not only left the
original bug in place but also introduced the problem that the QP
number bitmap would get full after a while.

Found by Dotan Barak of Mellanox.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

When mlx4_buf_free() is called from the error path of
mlx4_buf_alloc(), it may be passed a buffer structure that does not
have all pages filled in.  Add a check for NULL to mlx4_buf_free() so
we avoid passing NULL to dma_free_coherent() (which will crash).
Signed-off-by: NAli Ayoub <ali@mellanox.co.il>
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Most drivers need to set length and offset as well, so may as well fold
those three lines into one.

Add sg_assign_page() for those two locations that only needed to set
the page, where the offset/length is set outside of the function context.
Signed-off-by: NJens Axboe <jens.axboe@oracle.com>

Signed-off-by: NJens Axboe <jens.axboe@oracle.com>

The current INIT_HCA firmware command timeout is sufficient for the
default number of resources (QPs, CQs, etc) being allocated, but if
the HCA profile is modified to increase the amount of resources, then
a spurious timeout is detected and HCA initialization fails.

Increase the timeout for the INIT_HCA command to 10 seconds, which
also brings it into line with all the other command timeouts.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

    
Commit 3d73c288 ("mlx4_core: Fix section mismatches") introduced a
stupid bug in device init: when some of mlx4_init_one() was split off
into __mlx4_init_one(), the call from the main mlx4_init_one()
function was back to mlx4_init_one() rather than to __mlx4_init_one(),
which leads to an obvious infinite loop if the function is every
called.
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

    
Commit ee49bd93 ("mlx4_core: Reset device when internal error is
detected") introduced some section mismatch problems when
CONFIG_HOTPLUG=n, because the error recovery code tears down and
reinitializes the device after everything is loaded, which ends up
calling into lots of code marked __devinit and __devexit from regular
.text.  Fix this by getting rid of these now-incorrect section
markers.
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Firmware commands are sent to the HCA by writing multiple words to a
command register block.  Access to this block of registers is
serialized with a mutex.  However, on large SGI systems writes to the
register block may be reordered within the system interconnect and
reach the HCA in a different order than they were issued (even with
the mutex).  Fix this by adding an mmiowb() before dropping the mutex.

This bug was observed with real workloads with the similar FW command
code in the mthca driver, and adding the mmiowb() as in commit
66547550 ("IB/mthca: Use mmiowb() to avoid firmware commands getting
jumbled up") was confirmed to fix the problems, so we should add the
same fix to mlx4.
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Increase the number of QPs allowed per multicast group from 8 to 56.
This allows for one QP per core on 16-core systems, which are now
quite common, and allows some space for future growth.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Implement FMRs for mlx4.  This is an adaptation of code from mthca.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NMichael S. Tsirkin <mst@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Write MTT entries directly to ICM from the driver (eliminating use of
WRITE_MTT command).  This reduces the number of FW commands needed to
register an MR by at least a factor of 2 and speeds up memory
registration significantly.  This code will also be used to implement
FMRs.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NMichael S. Tsirkin <mst@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Everything that uses caps.reserved_mtts expects it to be a count of MTT
segments, not MTT entries.  So convert the value that the FW gives us to
a count of segments.
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Taking ilog2(dev->caps.reserved_mtts) to find out the order to pass to
the MTT buddy allocator will do the wrong thing if reserved_mtts is ever
not a power of 2.  Be safe and use fls(dev->caps.reserved_mtts - 1).
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Enable having ICM tables in coherent memory, and use coherent memory
for the dMPT table.  This will allow writing MPT entries for MRs both
via the SW2HW_MPT command and also directly by the driver for FMR
remapping without needing to flush or worry about cacheline boundaries.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NMichael S. Tsirkin <mst@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

display the following device information under /sys/class/infiniband/mlx4_X:
board_id, fw_ver, hw_rev, hca_type.

This patch makes this information available to userspace utilities
such as ibstat and ibv_devinfo.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

The SRC ("scalable RC") transport has been renamed to XRC ("extended 
RC"), to avoid having an abbreviation that is so easily confused with an 
abbreviation for "source."  Update the HCA capability decoding output to 
use the new name.
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

mlx4_srq_query() returns a big-endian 16-bit value through an int *,
which screws up sparse checking.  Fix this so that a CPU-endian value
is returned.
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Recover from MSI-X errors by automatically falling back on regular
interrupt, instead of asking the user to do this manually.  This makes
it possible to enable MSI-X by default, and will make it possible to
get rid of the msi_x module option in the future.
Signed-off-by: NMichael S. Tsirkin <mst@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Special QPs are not allocated using the regular QP number bitmap, so
when they are destroyed, their QP number should not be freed in the
bitmap.

Found by Dotan Barak of Mellanox.
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Rename GO_BIT_TIMEOUT to GO_BIT_TIMEOUT_MSECS for clarity, and
actually use it as the go bit timeout (instead of having the define
but then ignoring it and using a hard-coded 10 * HZ for the actual
timeout).
Signed-off-by: NDotan Barak <dotanb@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Put a 1000 msec delay after resetting the device before attempting to
do config cycles on it.  Not waiting causes system hangs on some
chipsets, e.g. Intel E7520, when the driver is loaded.
Signed-off-by: NEli Cohen <eli@mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

mlx4_mr_alloc() doesn't actually allocate mr (it just initializes the
pointer that the caller passes in), so it shouldn't free it if an
error occurs.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

    
The FW command token is currently only updated on a command completion
event. This means that on command timeout, the same token will be
reused for new command, which results in a mess if the timed out
command *does* eventually complete.

This is the same change as the patch for mthca from Michael
S. Tsirkin <mst@dev.mellanox.co.il> that was just merged.  It seems
sensible to avoid gratuitous differences in FW command processing
between mthca and mlx4.
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Change the maximum number of outstanding RDMA reads allowed as a
target from 4 to 16 to per QP.  This allows RDMA read operations to
pipeline better.

Pointed out by Dotan Barak and Sagi Rotem.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Reset the device when an internal error is detected.

Also, detect errors by polling the error buffer rather than using
interrupts.  This is more robust and doesn't depend on MSI-X.  Remove
the old interrupt handler entirely, since we don't want to support two
mechanisms for detecting internal errors.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Signed-off-by: NDotan Barak <dotanb@mellanox.co.il>
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Get the maximum message size from the device capabilities returned
from the QUERY_DEV_CAP firmware command, rather than hard-coding 2 GB.
Signed-off-by: NDotan Barak <dotanb@dev.mellanox.co.il>
Signed-off-by: NMichael S. Tsirkin <mst@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

mlx4.h uses struct mutex, so although <linux/mutex.h> seems to be pulled in
indirectly by one of the headers it includes, the right thing to do is
to include <linux/mutex.h> directly.
Signed-off-by: NMichael S. Tsirkin <mst@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Recent gcc versions emit warnings when unsigned variables are compared < 0 or >= 0.
Signed-off-by: NBill Nottingham <notting@redhat.com>
Signed-off-by: NJeff Garzik <jeff@garzik.org>

Add new IDs for PCIe gen2 devices.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>