The powerpc iommu uses a hardcoded page size of 4K. This patch changes
the name of the IOMMU_PAGE_* macros to reflect the hardcoded values. A
future patch will use the existing names to support dynamic page
sizes.
Signed-off-by: NAlistair Popple <alistair@popple.id.au>
Signed-off-by: NAlexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

The current implementation of IOMMU on sPAPR does not use iommu_ops
and therefore does not call IOMMU API's bus_set_iommu() which
1) sets iommu_ops for a bus
2) registers a bus notifier
Instead, PCI devices are added to IOMMU groups from
subsys_initcall_sync(tce_iommu_init) which does basically the same
thing without using iommu_ops callbacks.

However Freescale PAMU driver (https://lkml.org/lkml/2013/7/1/158)
implements iommu_ops and when tce_iommu_init is called, every PCI device
is already added to some group so there is a conflict.

This patch does 2 things:
1. removes the loop in which PCI devices were added to groups and
adds explicit iommu_add_device() calls to add devices as soon as they get
the iommu_table pointer assigned to them.
2. moves a bus notifier to powernv code in order to avoid conflict with
the notifier from Freescale driver.

iommu_add_device() and iommu_del_device() are public now.
Signed-off-by: NAlexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Signed-off-by: NAnton Blanchard <anton@samba.org>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

As a part of pseries_idle backend driver cleanup to make
the code common to both pseries and powernv platforms, it
is necessary to move the backend-driver code to drivers/cpuidle.

As a pre-requisite for that, it is essential to move plpar_wrapper.h
to include/asm.
Signed-off-by: NDeepthi Dharwar <deepthi@linux.vnet.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

We pass dma_window to of_parse_dma_window as a void * and then
run through hoops to cast it back to a u32 array. In the process
we lose endian annotation.

Simplify it by just passing a __be32 * down.
Signed-off-by: NAnton Blanchard <anton@samba.org>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

The enables VFIO on the pSeries platform, enabling user space
programs to access PCI devices directly.
Signed-off-by: NAlexey Kardashevskiy <aik@ozlabs.ru>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: NPaul Mackerras <paulus@samba.org>
Acked-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Given a PCI device with multiple functions in a DDW capable slot, the
following situation can be encountered: When the first function sets a
64-bit DMA mask, enable_ddw() will be called and we can fail to properly
configure DDW (the most common reason being the new DMA window's size is
not large enough to map all of an LPAR's memory). With the recent
changes to DDW, we remove the base window in order to determine if the
new window is of sufficient size to cover an LPAR's memory. We correctly
replace the base window if we find that not to be the case. However,
once we go through and re-configured 32-bit DMA via the IOMMU, the next
function of the adapter will go through the same process. And since DDW
is a characteristic of the slot itself, we are most likely going to fail
again. But to determine we are going to fail the second slot, we again
remove the base window -- but that is now in-use by the first
function/driver, which might be issuing I/O already.

To close this window, keep a list of all the failed struct device_nodes
that have failed to configure DDW. If the current device_node is in that
list, just fail out immediately and fall back to 32-bit DMA without
doing any DDW manipulation.
Signed-off-by: NNishanth Aravamudan <nacc@linux.vnet.ibm.com>
Signed-off-by: NMichael Ellerman <michael@ellerman.id.au>

pseries/iommu: remove DDW on kexec

We currently insert a property in the device-tree when we successfully
configure DDW for a given slot. This was meant to be an optimization to
speed up kexec/kdump, so that we don't need to make the RTAS calls again
to re-configured DDW in the new kernel.

However, we end up tripping a plpar_tce_stuff failure on kexec/kdump
because we unconditionally parse the ibm,dma-window property for the
node at bus/dev setup time. This property contains the 32-bit DMA window
LIOBN, which is distinct from the DDW window's. We pass that LIOBN (via
iommu_table_init -> iommu_table_clear -> tce_free ->
tce_freemulti_pSeriesLP) to plpar_tce_stuff, which fails because that
32-bit window is no longer present after
25ebc45b ("powerpc/pseries/iommu: remove
default window before attempting DDW manipulation").

I believe the simplest, easiest-to-maintain fix is to just change our
initcall to, rather than detecting and updating the new kernel's DDW
knowledge, just remove all DDW configurations. When the drivers
re-initialize, we will set everything back up as it was before.
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

The parameter is unused, and complicates a following fix. Just remove
it.
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

There are now two kinds of DMA windows that might be presented by
PowerVM DDW support -- huge windows (that can map all of system memory
regardless of the LPAR configuration) and non-huge windows (which
can't). They are implemented slightly differently in PowerVM, and thus
have different characteristics. The most obvious is that slot isolate
doesn't clear the TCEs/window for us with non-huge windows. Thus, when a
DLPAR operation occurs on a slot using a non-huge window, TCEs are still
present (the notifier chain doesn't currently remove them explicitly)
and the DLPAR fails. Fix this by calling remove_ddw() first, which will
unmap the DDW TCEs.

Note: a corresponding change to drmgr is needed to actually successfully
DLPAR, such that the device-tree update (which causes the notifier chain
to fire) occurs before slot isolate.
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

tce_clearrange_multi_pSeriesLP is attempting to iterate over all TCEs in
a given range. However, is it not advancing the dma_offset value passed
to plpar_tce_stuff via the next value. This prevents DLPAR from
completing, because TCEs are still present at slot isolation time.
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Rename the prom_*_property routines of the generic OF code to of_*_property.
This brings them in line with the naming used by the rest of the OF code.
Signed-off-by: NNathan Fontenot <nfont@linux.vnet.ibm.com>
Acked-by: NGeoff Levand <geoff@infradead.org>
Acked-by: NRob Herring <rob.herring@calxeda.com>
Acked-by: NGrant Likely <grant.likely@secretlab.ca>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

This patch moves the notification chain for updates to the device tree
from the powerpc/pseries code to the base OF code. This makes this
functionality available to all architectures.

Additionally the notification chain is updated to allow notifications
for property add/remove/update. To make this work a pointer to a new
struct (of_prop_reconfig) is passed to the routines in the notification chain.
The of_prop_reconfig property contains a pointer to the node containing the
property and a pointer to the property itself. In the case of property
updates, the property pointer refers to the new property.
Signed-off-by: NNathan Fontenot <nfont@linux.vnet.ibm.com>
Acked-by: NRob Herring <rob.herring@calxeda.com>
Acked-by: NGrant Likely <grant.likely@secretlab.ca>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

It's empty now, apart from other includes.

Fixup a few files that were getting things via this header.
Signed-off-by: NMichael Ellerman <michael@ellerman.id.au>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

These days they are just __va() and __pa() respectively.
Signed-off-by: NMichael Ellerman <michael@ellerman.id.au>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

The pattern (np ? np->full_name : "<none>") is rather common in the
kernel, but can also make for quite long lines.  This patch adds a new
inline function, of_node_full_name() so that the test for a valid node
pointer doesn't need to be open coded at all call sites.
Signed-off-by: NGrant Likely <grant.likely@secretlab.ca>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: NRob Herring <rob.herring@calxeda.com>

tce_buildmulti_pSeriesLP uses a per cpu page to communicate with the
hypervisor. We currently rely on the IOMMU table spinlock but
subsequent patches will be removing that so disable interrupts
around all accesses of tce_page.
Signed-off-by: NAnton Blanchard <anton@samba.org>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

An upcoming release of firmware will add DDW extensions, in particular
an API to "reset" the DMA window to the original configuration (32-bit,
2GB in size). With that API available, we can safely remove the default
window, increasing the resources available to firmware for creation of
larger windows for the slot in question -- if we encounter an error, we
can use the new API to reset the state of the slot.

Further, this same release of firmware will make it a hard requirement
for OSes to release the existing window before any other windows will be
shown as available, to avoid conflicts in addressing between the two
windows.

In anticipation of these changes, always remove the default window
before we do any DDW manipulations.
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

The following added support for powernv but broke pseries/BML:
 1f1616e8 powerpc/powernv: Add TCE SW invalidation support

TCE_PCI_SW_INVAL was split into FREE and CREATE flags but the tests in
the pseries code were not updated to reflect this.
Signed-off-by: NMichael Neuling <mikey@neuling.org>
cc: stable@kernel.org [v3.3+]
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

The pattern (np ? np->full_name : "<none>") is rather common in the
kernel, but can also make for quite long lines.  This patch adds a new
inline function, of_node_full_name() so that the test for a valid node
pointer doesn't need to be open coded at all call sites.
Signed-off-by: NGrant Likely <grant.likely@secretlab.ca>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Thomas Gleixner <tglx@linutronix.de>

The patch removes the eeh information from pci_dn since the eeh
device (struct eeh_dev) already contained those information and
the copy in pci_dn is no longer used except for the pseries iommu
mapping code, which we change to retrieve the PE address from eeh
device instead.
Signed-off-by: NGavin Shan <shangw@linux.vnet.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

This is used for newer IO Hubs such as p7IOC.
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Some pseries IOMMUs cache TCEs but don't snoop when the TCEs are changed
in memory, hence we need manually invalidate in software.

This adds code to do the invalidate.  It keys off a device tree property
to say where the to do the MMIO for the invalidate and some information
on what the format of the invalidate including some magic routing info.

it_busno get overloaded with this magic routing info and it_index with
the MMIO address for the invalidate command.

This then gets hooked into the building and freeing of TCEs.

This is only useful on bare metal pseries.  pHyp takes care of this when
virtualised.

Based on patch from Milton with cleanups from Mikey.
Signed-off-by: NMilton Miller <miltonm@bga.com>
Signed-off-by: NMichael Neuling <mikey@neuling.org>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

They are getting it through device.h --> module.h path, but we want
to clean that up.  This is a sample of what will happen if we don't:

  pseries/iommu.c: In function 'tce_build_pSeriesLP':
  pseries/iommu.c:136: error: implicit declaration of function 'show_stack'

  pseries/eeh.c: In function 'eeh_token_to_phys':
  pseries/eeh.c:359: error: 'init_mm' undeclared (first use in this function)

  pseries/eeh_event.c: In function 'eeh_event_handler':
  pseries/eeh_event.c:63: error: implicit declaration of function 'daemonize'
  pseries/eeh_event.c:64: error: implicit declaration of function 'set_current_state'
  pseries/eeh_event.c:64: error: 'TASK_INTERRUPTIBLE' undeclared (first use in this function)
  pseries/eeh_event.c:64: error: (Each undeclared identifier is reported only once
  pseries/eeh_event.c:64: error: for each function it appears in.)
  pseries/eeh_event.c: In function 'eeh_thread_launcher':
  pseries/eeh_event.c:109: error: 'CLONE_KERNEL' undeclared (first use in this function)

  hotplug-cpu.c: In function 'pseries_mach_cpu_die':
  hotplug-cpu.c:115: error: implicit declaration of function 'idle_task_exit'

  kernel/swsusp_64.c: In function 'do_after_copyback':
  kernel/swsusp_64.c:17: error: implicit declaration of function 'touch_softlockup_watchdog'

  cell/spufs/context.c: In function 'alloc_spu_context':
  cell/spufs/context.c:60: error: implicit declaration of function 'get_task_mm'
  cell/spufs/context.c:60: warning: assignment makes pointer from integer without a cast
  cell/spufs/context.c: In function 'spu_forget':
  cell/spufs/context.c:127: error: implicit declaration of function 'mmput'

  pasemi/dma_lib.c: In function 'pasemi_dma_stop_chan':
  pasemi/dma_lib.c:332: error: implicit declaration of function 'cond_resched'

  sysdev/fsl_lbc.c: In function 'fsl_lbc_ctrl_irq':
  sysdev/fsl_lbc.c:247: error: 'TASK_NORMAL' undeclared (first use in this function)

Add in sched.h so these get the definitions they are looking for.
Signed-off-by: NPaul Gortmaker <paul.gortmaker@windriver.com>

At this point, window has not been stored anywhere, so it has to be freed
before leaving the function.

A simplified version of the semantic match that finds this problem is as
follows: (http://coccinelle.lip6.fr/)

// <smpl>
@exists@
local idexpression x;
statement S,S1;
expression E;
identifier fl;
expression *ptr != NULL;
@@

x = \(kmalloc\|kzalloc\|kcalloc\)(...);
...
if (x == NULL) S
<... when != x
     when != if (...) { <+...kfree(x)...+> }
     when any
     when != true x == NULL
x->fl
...>
(
if (x == NULL) S1
|
if (...) { ... when != x
               when forall
(
 return \(0\|<+...x...+>\|ptr\);
|
* return ...;
)
}
)
// </smpl>
Signed-off-by: NJulia Lawall <julia@diku.dk>
Acked-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Now that the generic code has dma_map_ops set, instead of having a
messy ifdef & if block in the base dma_get_required_mask hook push
the computation into the dma ops.

If the ops fails to set the get_required_mask hook default to the
width of dma_addr_t.

This also corrects ibmbus ibmebus_dma_supported to require a 64
bit mask.  I doubt anything is checking or setting the dma mask on
that bus.
Signed-off-by: NMilton Miller <miltonm@bga.com>
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-kernel@vger.kernel.org
Cc: benh@kernel.crashing.org
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

The hook dma_get_required_mask is supposed to return the mask required
by the platform to operate efficently.  The generic version of
dma_get_required_mask in driver/base/platform.c returns a mask based
only on max_pfn.  However, this is likely too big for iommu systems
and could be too small for platforms that require a dma offset or have
a secondary window at a high offset.

Override the default, provide a hook in ppc_md used by pseries lpar and
cell, and provide the default answer based on memblock_end_of_DRAM(),
with hooks for get_dma_offset, and provide an implementation for iommu
that looks at the defined table size.  Coverting from the end address
to the required bit mask is based on the generic implementation.

The need for this was discovered when the qla2xxx driver switched to
64 bit dma then reverted to 32 bit when dma_get_required_mask said
32 bits was sufficient.
Signed-off-by: NMilton Miller <miltonm@bga.com>
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-kernel@vger.kernel.org
Cc: benh@kernel.crashing.org
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

When using a property refering to the availibily of dynamic dma windows
call it ddw_avail not ddr_avail.

dupe_ddw_if_already_created does not dupilcate anything, it only finds
and reuses the windows we already created, so rename it to
find_existing_ddw.  Also, it does not need the pci device node, so
remove that argument.
Signed-off-by: NMilton Miller <miltonm@bga.com>
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Move the discovery of windows previously setup from when the pci driver
calls set_dma_mask to an arch_initcall.

When kexecing into a kernel with dynamic dma windows allocated, we need
to find the windows early so that memory hot remove will be able to
delete the tces mapping the to be removed memory and memory hotplug add
will map the new memory into the window.  We should not wait for the
driver to be loaded and the device to be probed.  The iommu init hooks
are before kmalloc is setup, so defer to arch_initcall.
Signed-off-by: NMilton Miller <miltonm@bga.com>
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

If we destroy the window, we need to remove the property recording that
we setup the window.  Otherwise the next kernel we kexec will be
confused.

Also we should remove the property if even if we don't find the
ibm,ddw-applicable window or if one of the property sizes is unexpected;
presumably these came from a prior kernel via kexec, and we will not be
maintaining the window with respect to memory hotplug.
Signed-off-by: NMilton Miller <miltonm@bga.com>
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Do not check dma supported until we have chosen the right dma ops.
Check that the device is pci before treating it as such.

Check the mask is supported by the selected dma ops before
committing it.

We only need to set iommu ops if it is not the current ops; this
avoids searching the tree for the iommu table unnecessarily.
Signed-off-by: NMilton Miller <miltonm@bga.com>
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Otherwise we get silent truncations.
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Cc: Anton Blanchard <anton@samba.org>
Cc: Milton Miller <miltonm@bga.com>
Cc: linuxppc-dev@ozlabs.org
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

When we swtich to direct dma ops, we set the dma data union to have the
dma offset.  When we switch back to iommu table ops because of a later
dma_set_mask, we need to restore the iommu table pointer. Without this
change, crashes have been observed on kexec where (for reasons still
being investigated) we fall back to a 32-bit dma mask on a particular
device and then panic because the table pointer is not valid.

The easiset way to find this value is to call
pci_dma_dev_setup_pSeriesLP which will search up the pci tree until it
finds the node with the table.
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Cc: Milton Miller <miltonm@bga.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Anton Blanchard <anton@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

When we kexec we look for a particular property added by the first
kernel, "linux,direct64-ddr-window-info", per-device where we already
have set up dynamic dma windows. The current code, though, wasn't
initializing the size of this property and thus when we kexec'd, we
would find the property but read uninitialized memory resulting in
garbage ddw values for the kexec'd kernel and panics. Fix this by
setting the size at enable_ddw() time and ensuring that the size of the
found property is valid at dupe_ddw_if_kexec() time.
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Fixes generated by 'codespell' and manually reviewed.
Signed-off-by: NLucas De Marchi <lucas.demarchi@profusion.mobi>

If firmware allows us to map all of a partition's memory for DMA on a
particular bridge, create a 1:1 mapping of that memory. Add hooks for
dealing with hotplug events. Dynamic DMA windows can use larger than the
default page size, and we use the largest one possible.
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

iommu_table_setparms_lpar needs either the phb or the subbusnumber
(not both), pass the phb to make it similar to iommu_table_setparms.

Note: In cases where a caller was passing bus->number previously to
iommu_table_setparms_lpar() rather than phb->bus->number, this can lead
to a different value in tbl->it_busno. The only example of this was the
removed pci_dma_dev_setup_pSeriesLP(), removed in "ppc/iommu: remove
unneeded pci_dma_dev_setup_pSeriesLP".

[BenH: You updated only one of the two callers. Fixed that for you]
Signed-off-by: NMilton Miller <miltonm@bga.com>
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

The block in pci_dma_dev_setup_pSeriesLP for dma_window == NULL can be
removed because we will only teminate the loop if we had already allocated
a iommu table for that node or we found a window.  While there may be
no window for the device, the intresting part is if we are reusing a
table or creating it for the first device under it.
Signed-off-by: NMilton Miller <miltonm@bga.com>
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

The device tree root is never a pci bus, and will not have a
PCI_DN(pdn), so the check for PCI_DN added in
650f7b3b makes the check for pdn->parent
redundant and it can be removed.
Signed-off-by: NMilton Miller <miltonm@bga.com>
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

The iommu_table pointer in the pci auxiliary struct of device_node has
not been used by the iommu ops since the dma refactor of
12d04eef, however this code still uses
it to find tables for dlpar. By only setting the PCI_DN iommu_table
pointer on nodes with dma window properties, we will be able to quickly
find the node for later checks, and can remove the table without looking
for the the dma window property on dlpar remove.
Signed-off-by: NMilton Miller <miltonm@bga.com>
Signed-off-by: NNishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>