In case of large queue pairs there is the possibillity of allocation
failures due to memory fragmentation when using kmalloc().  To ensure
the memory is allocated even if kmalloc() can not find chunks which
are big enough, we fall back to allocating the memory with vmalloc().
Signed-off-by: NStefan Roscher <stefan.roscher@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

To improve performance of driver resource allocation, replace
vmalloc() calls with kmalloc().
Signed-off-by: NStefan Roscher <stefan.roscher@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

The base versions handle constant folding just fine, use them
directly.  The replacements are OK in the include/ files as they are
not exported to userspace so we don't need the __ prefixed versions.

This patch does not affect code generation at all.
Signed-off-by: NHarvey Harrison <harvey.harrison@gmail.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

ehca_plpar_hcall9() takes an unsigned long array, so make all callers
pass that in.  This fixes warnings introduced by commit fe333321
("powerpc: Change u64/s64 to a long long integer type"), which changed
u64 from unsigned long to unsigned long long.
Signed-off-by: NStephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Commit fe333321 ("powerpc: Change u64/s64 to a long long integer
type") changed u64 from unsigned long to unsigned long long, which
means that printk formats for printing u64 values should use "ll"
instead of "l" to avoid warnings.  Fix all the places affected by this
in ehca.
Signed-off-by: NStephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

The flags argument to spin_lock_irqsave() should really be unsigned
long.  This will also help prevent some warnings when we change u64 to
unsigned long long.
Signed-off-by: NStephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Impact: cleanup

We're moving from handing around cpumask_t's to handing around struct
cpumask *'s.  cpus_*, cpumask_t and cpu_*_map are deprecated: convert
to cpumask_*, cpu_*_mask.
Signed-off-by: NRusty Russell <rusty@rustcorp.com.au>
Acked-by: NHoang-Nam Nguyen <hnguyen@de.ibm.com>
Tested-by: NHoang-Nam Nguyen <hnguyen@de.ibm.com>
Cc: Christoph Raisch <raisch@de.ibm.com>

Impact: cleanup

In future, accessing cpu numbers beyond nr_cpu_ids (the runtime limit)
will be undefined.  We can avoid future problems by using
for_each_online_cpu() here.
Signed-off-by: NRusty Russell <rusty@rustcorp.com.au>
Acked-by: NHoang-Nam Nguyen <hnguyen@de.ibm.com>
Tested-by: NHoang-Nam Nguyen <hnguyen@de.ibm.com>
Cc: Christoph Raisch <raisch@de.ibm.com>

vpage is checked not to be NULL just after it is initialized at the
beginning of each loop iteration.

A simplified version of the semantic patch that makes this change is
as follows: (http://www.emn.fr/x-info/coccinelle/)

// <smpl>
@r exists@
local idexpression x;
expression E;
position p1,p2;
@@

if (x@p1 == NULL || ...) { ... when forall
   return ...; }
... when != \(x=E\|x--\|x++\|--x\|++x\|x-=E\|x+=E\|x|=E\|x&=E\|&x\)
(
x@p2 == NULL
|
x@p2 != NULL
)

// another path to the test that is not through p1?
@s exists@
local idexpression r.x;
position r.p1,r.p2;
@@

... when != x@p1
(
x@p2 == NULL
|
x@p2 != NULL
)

@fix depends on !s@
position r.p1,r.p2;
expression x,E;
statement S1,S2;
@@

(
- if ((x@p2 != NULL) || ...)
  S1
|
- if ((x@p2 == NULL) && ...) S1
|
- BUG_ON(x@p2 == NULL);
)
// </smpl>
Signed-off-by: NJulia Lawall <julia@diku.dk>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

With the latest flush error completion patch we introduced modulus
operation to calculate the next index within a qmap.  Based on
comments from other mailing lists we decided to optimize this
operation by using an addition and an if-statement instead of modulus,
even though this is on the error path.
Signed-off-by: NStefan Roscher <stefan.roscher@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

shca_list_lock is taken from softirq context in ehca_poll_eqs, so we
need to lock IRQ safe elsewhere.  Found by lockdep.
Signed-off-by: NMichael Ellerman <michael@ellerman.id.au>
Signed-off-by: NJoachim Fenkes <fenkes@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

This fix enables ehca device driver to generate flush work completions
even if the application doesn't request completions for all work
requests. The current implementation of ehca will generate flush work
completions for the wrong work requests if an application uses non
signaled work completions.
Signed-off-by: NStefan Roscher <stefan.roscher@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

The error message printed when the eHCA driver prevents memory hotplug
is misleading -- the user might think that hot-removing the lhca,
hotplugging memory, then hot-adding the lhca again will work, but it
actually doesn't.
Signed-off-by: NJoachim Fenkes <fenkes@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

If the initialization of a special QP (e.g. AQP1) fails due to a
software timeout, we have to remove the reference to that special QP
struct from the port struct to stop the driver from accessing the QP,
since it will be/has been destroyed by the caller, eg in this case
ib_mad.
Signed-off-by: NStefan Roscher <stefan.roscher@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Since the ehca device driver does not support dynamic memory add and
remove operations, the driver must explicitly reject such requests in
order to prevent unpredictable behaviors related to existing memory
regions that cover all of memory being used by InfiniBand protocols in
the kernel.

The solution (for now at least) is to add a memory notifier to the
ehca device driver and if a request for dynamic memory add or remove
comes in, ehca will always reject it.  The user can add or remove
memory by hot-removing the ehca adapter, performing the memory
operation, and then hot-adding the ehca adapter back.
Signed-off-by: NStefan Roscher <stefan.roscher@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Because ehca adapters can differ in the maximum number of QPs and CQs
we have to save the maximum number of these ressources per adapter and
not globally per ehca driver. This fix introduces 2 new members to the
shca structure to store the maximum value for QPs and CQs per adapter.

The module parameters are now used as initial values for those
variables.  If a user selects an invalid number of CQs or QPs we don't
print an error any longer, instead we will inform the user with a
warning and set the values to the respective maximum supported by the
HW.
Signed-off-by: NStefan Roscher <stefan.roscher@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

This patch prevents a UC QP to be created attached to an SRQ, since
current firmware does not support this feature.
Signed-off-by: NMichael Faath <micfaath@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

When a QP goes into error state, it is required that CQ entries with a
flush error status are delivered to the application for any
outstanding work requests.  eHCA does not do this in hardware, so this
patch adds software flush CQE generation to the ehca driver.

Whenever a QP gets into error state, it is added to the QP error list
of its respective CQ.  If the error QP list of a CQ is not empty,
poll_cq() generates flush CQEs before polling the actual CQ.
Signed-off-by: NAlexander Schmidt <alexs@linux.vnet.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

This patch lets the files using linux/version.h match the files that
#include it.
Signed-off-by: NAdrian Bunk <bunk@kernel.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Under rare circumstances, the ehca hardware might erroneously generate
two CQEs for the same WQE, which is not compliant to the IB spec and
will cause unpredictable errors like memory being freed twice. To
avoid this problem, the driver needs to detect the second CQE and
discard it.

For this purpose, introduce an array holding as many elements as the
SQ of the QP, called sq_map. Each sq_map entry stores a "reported"
flag for one WQE in the SQ. When a work request is posted to the SQ,
the respective "reported" flag is set to zero. After the arrival of a
CQE, the flag is set to 1, which allows to detect the occurence of a
second CQE.

The mapping between WQE / CQE and the corresponding sq_map element is
implemented by replacing the lowest 16 Bits of the wr_id with the
index in the queue map. The original 16 Bits are stored in the sq_map
entry and are restored when the CQE is passed to the application.
Signed-off-by: NAlexander Schmidt <alexs@linux.vnet.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

The idr_find() function may fail when trying to get the QP that is
associated with a CQE, e.g. when a QP has been destroyed between the
generation of a CQE and the poll request for it.  In consequence, the
return value of idr_find() must be checked and the CQE must be
discarded when the QP cannot be found.
Signed-off-by: NAlexander Schmidt <alexs@linux.vnet.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

When the ehca driver detects an invalid opcode in a CQE, it currently
passes the CQE to the application and returns with success. This patch
changes the CQE handling to discard CQEs with invalid opcodes and to
continue reading the next CQE from the CQ.
Signed-off-by: NAlexander Schmidt <alexs@linux.vnet.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Rename the "poll_cq_one_read_cqe" goto label to what it actually does,
namely "repoll".
Signed-off-by: NAlexander Schmidt <alexs@linux.vnet.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Since the introduction of the port auto-detect mode for ehca, calls to
modify_qp() may be cached in the device driver when the ports are not
activated yet. When a modify_qp() call is cached, the qp state remains
untouched until the port is activated, which will leave the qp in the
reset state. In the reset state, however, it is not allowed to post SQ
WQEs, which confuses applications like ib_mad.

The solution for this problem is to immediately set the qp state as
requested by modify_qp(), even when the call is cached.
Signed-off-by: NAlexander Schmidt <alexs@linux.vnet.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

from include/asm-powerpc.  This is the result of a

mkdir arch/powerpc/include/asm
git mv include/asm-powerpc/* arch/powerpc/include/asm

Followed by a few documentation/comment fixups and a couple of places
where <asm-powepc/...> was being used explicitly.  Of the latter only
one was outside the arch code and it is a driver only built for powerpc.
Signed-off-by: NStephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

The pd->lock mutex is released on a successful return, so it should be
released on an error return as well.

The semantic patch that makes this change is as follows:
(http://www.emn.fr/x-info/coccinelle/)

// <smpl>
@@
expression l;
@@

mutex_lock(l);
... when != mutex_unlock(l)
    when any
    when strict
(
if (...) { ... when != mutex_unlock(l)
+   mutex_unlock(l);
    return ...;
}
|
mutex_unlock(l);
)
// </smpl>
Signed-off-by: NJulia Lawall <julia@diku.dk>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Some firmware versions report a Local CA ACK Delay of 0.  In that
case, return a more sensible default value of 12 (-> 16 msec) instead.
Signed-off-by: NJoachim Fenkes <fenkes@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Certain firmware versions sometimes cause spurious PATH_MIG events to
occur during QP creation.  Filter these events by making sure PATH_MIG
events are only handed down when they actually make sense (i.e. when
the QP has been armed at least once).
Signed-off-by: NJoachim Fenkes <fenkes@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

This gives ehca an autogenerated modalias and therefore enables automatic loading.
Signed-off-by: NJoachim Fenkes <fenkes@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

During corner case testing, we noticed that some versions of ehca do
not properly transition to interrupt done in special load situations.
This can be resolved by periodically triggering EOI through H_EOI, if
EQEs are pending.
Signed-off-by: NStefan Roscher <stefan.roscher@de.ibm.com>
Acked-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Signed-off-by: NJoachim Fenkes <fenkes@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

This patch adds support for the IB "base memory management extension"
(BMME) and the equivalent iWARP operations (which the iWARP verbs
mandates all devices must implement).  The new operations are:

 - Allocate an ib_mr for use in fast register work requests.

 - Allocate/free a physical buffer lists for use in fast register work
   requests.  This allows device drivers to allocate this memory as
   needed for use in posting send requests (eg via dma_alloc_coherent).

 - New send queue work requests:
   * send with remote invalidate
   * fast register memory region
   * local invalidate memory region
   * RDMA read with invalidate local memory region (iWARP only)

Consumer interface details:

 - A new device capability flag IB_DEVICE_MEM_MGT_EXTENSIONS is added
   to indicate device support for these features.

 - New send work request opcodes IB_WR_FAST_REG_MR, IB_WR_LOCAL_INV,
   IB_WR_RDMA_READ_WITH_INV are added.

 - A new consumer API function, ib_alloc_mr() is added to allocate
   fast register memory regions.

 - New consumer API functions, ib_alloc_fast_reg_page_list() and
   ib_free_fast_reg_page_list() are added to allocate and free
   device-specific memory for fast registration page lists.

 - A new consumer API function, ib_update_fast_reg_key(), is added to
   allow the key portion of the R_Key and L_Key of a fast registration
   MR to be updated.  Consumers call this if desired before posting
   a IB_WR_FAST_REG_MR work request.

Consumers can use this as follows:

 - MR is allocated with ib_alloc_mr().

 - Page list memory is allocated with ib_alloc_fast_reg_page_list().

 - MR R_Key/L_Key "key" field is updated with ib_update_fast_reg_key().

 - MR made VALID and bound to a specific page list via
   ib_post_send(IB_WR_FAST_REG_MR)

 - MR made INVALID via ib_post_send(IB_WR_LOCAL_INV),
   ib_post_send(IB_WR_RDMA_READ_WITH_INV) or an incoming send with
   invalidate operation.

 - MR is deallocated with ib_dereg_mr()

 - page lists dealloced via ib_free_fast_reg_page_list().

Applications can allocate a fast register MR once, and then can
repeatedly bind the MR to different physical block lists (PBLs) via
posting work requests to a send queue (SQ).  For each outstanding
MR-to-PBL binding in the SQ pipe, a fast_reg_page_list needs to be
allocated (the fast_reg_page_list is owned by the low-level driver
from the consumer posting a work request until the request completes).
Thus pipelining can be achieved while still allowing device-specific
page_list processing.

The 32-bit fast register memory key/STag is composed of a 24-bit index
and an 8-bit key.  The application can change the key each time it
fast registers thus allowing more control over the peer's use of the
key/STag (ie it can effectively be changed each time the rkey is
rebound to a page list).
Signed-off-by: NSteve Wise <swise@opengridcomputing.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Signed-off-by: NJoachim Fenkes <fenkes@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Change references from for_each_cpu_mask to for_each_cpu_mask_nr
where appropriate
Reviewed-by: NPaul Jackson <pj@sgi.com>
Reviewed-by: NChristoph Lameter <clameter@sgi.com>
Signed-off-by: NMike Travis <travis@sgi.com>
Signed-off-by: NIngo Molnar <mingo@elte.hu>
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>

This is necessary because, in a multicore environment, a race between
uverbs async handler and destroy QP could occur.

Signed-off-by: Stefan Roscher <stefan.roscher at de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Also remove duplicate assignment of local_ca_ack_delay and change
min_t check for local_ca_ack_delay to u8 instead of int.

Signed-off-by: Stefan Roscher <stefan.roscher at de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

If a lot of QPs fall into Error state at once and the EQ of the
respective HCA is too small, it might overrun, causing the eHCA driver
to stop processing completion events and calling the application's
completion handlers, effectively causing traffic to stop.

Fix this by limiting available QPs and CQs to a customizable max
count, and determining EQ size based on these counts and a worst-case
assumption.
Signed-off-by: NStefan Roscher <stefan.roscher@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

ehca_create_eq() was assigning a signed return value to an unsiged
local variable and then checking if the variable was < 0, which meant
that errors were always ignored.  Fix this by using one variable for
signed integer return values and another for u64 hcall return values.

Bug originally found by Roel Kluin <12o3l@tiscali.nl>.
Signed-off-by: NHoang-Nam Nguyen <hnguyen@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>

Add a new parameter, dmasync, to the ib_umem_get() prototype.  Use dmasync = 1
when mapping user-allocated CQs with ib_umem_get().
Signed-off-by: NArthur Kepner <akepner@sgi.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: Jes Sorensen <jes@sgi.com>
Cc: Randy Dunlap <randy.dunlap@oracle.com>
Cc: Roland Dreier <rdreier@cisco.com>
Cc: James Bottomley <James.Bottomley@HansenPartnership.com>
Cc: David Miller <davem@davemloft.net>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Grant Grundler <grundler@parisc-linux.org>
Cc: Michael Ellerman <michael@ellerman.id.au>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Signed-off-by: NJoachim Fenkes <fenkes@de.ibm.com>
Signed-off-by: NRoland Dreier <rolandd@cisco.com>