.fault now can retry.  The retry can break state machine of .fault.  In
filemap_fault, if page is miss, ra->mmap_miss is increased.  In the second
try, since the page is in page cache now, ra->mmap_miss is decreased.  And
these are done in one fault, so we can't detect random mmap file access.

Add a new flag to indicate .fault is tried once.  In the second try, skip
ra->mmap_miss decreasing.  The filemap_fault state machine is ok with it.

I only tested x86, didn't test other archs, but looks the change for other
archs is obvious, but who knows :)
Signed-off-by: NShaohua Li <shaohua.li@fusionio.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Implement an interval tree as a replacement for the VMA prio_tree.  The
algorithms are similar to lib/interval_tree.c; however that code can't be
directly reused as the interval endpoints are not explicitly stored in the
VMA.  So instead, the common algorithm is moved into a template and the
details (node type, how to get interval endpoints from the node, etc) are
filled in using the C preprocessor.

Once the interval tree functions are available, using them as a
replacement to the VMA prio tree is a relatively simple, mechanical job.
Signed-off-by: NMichel Lespinasse <walken@google.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Hillf Danton <dhillf@gmail.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Woodhouse <dwmw2@infradead.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Signed-off-by: NHiroshi Doyu <hdoyu@nvidia.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

With ixp2xxx removed, there are no platforms that define arch_is_coherent,
so the last occurrences of arch_is_coherent can be removed. Any new
platform with coherent i/o should use coherent dma mapping functions.
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Remove arch_is_coherent() from iommu dma ops and implement separate
coherent ops functions.
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

arch_is_coherent is problematic as it is a global symbol. This
doesn't work for multi-platform kernels or platforms which can support
per device coherent DMA.

This adds arm_coherent_dma_ops to be used for devices which connected
coherently (i.e. to the ACP port on Cortex-A9 or A15). The arm_dma_ops
are modified at boot when arch_is_coherent is true.
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

With many IOMMU'able devices, console gets noisy.

Tegra30 has a few dozen of IOMMU'able devices.
Signed-off-by: NHiroshi Doyu <hdoyu@nvidia.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Skip unnecessary operations if order == 0. A little bit easier to
read.
Signed-off-by: NHiroshi Doyu <hdoyu@nvidia.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

arm: Add ARM ERRATA 775420 workaround

Workaround for the 775420 Cortex-A9 (r2p2, r2p6,r2p8,r2p10,r3p0) erratum.
In case a date cache maintenance operation aborts with MMU exception, it
might cause the processor to deadlock. This workaround puts DSB before
executing ISB if an abort may occur on cache maintenance.

Based on work by Kouei Abe and feedback from Catalin Marinas.
Signed-off-by: NKouei Abe <kouei.abe.cp@rms.renesas.com>
[ horms@verge.net.au: Changed to implementation
  suggested by catalin.marinas@arm.com ]
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NSimon Horman <horms@verge.net.au>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This allows /proc/vmallocinfo to show the physical address for
ioremap mappings.
Acked-by: NNicolas Pitre <nico@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

When either of __alloc_from_contiguous or __alloc_remap_buffer fails
to provide a valid pointer, allocated memory is freed up and an error
is returned. 'pages' was however not freed before returning error.

Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NSachin Kamat <sachin.kamat@linaro.org>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Fix this harmless build warning:

arch/arm/mm/alignment.c: In function 'do_alignment':
arch/arm/mm/alignment.c:749:21: warning: 'offset.un' may be used uninitialized in this function

This is caused by the compiler not being able to properly analyse the
code to prove that offset.un is assigned in every case.  The case it
struggles with is where we assign the handler from the Thumb parser -
do_alignment_t32_to_handler().  As this starts by zeroing this variable
via a pointer, move it into the calling function.  This fixes the
warning.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

There is a bug if l2x0 controller has been enabled when calling
l2x0_init, the aux ctrl register will not be saved in l2x0_saved_regs.
Therefore we will use uninitialized l2x0_saved_regs.aux_ctrl for
resuming later.

In this patch, the aux ctrl value is read and saved after it is
initialized. So we have the real value being set for resuming.

Link: http://lkml.kernel.org/r/1336046857-24133-1-git-send-email-ylmao@marvell.comSigned-off-by: NYilu Mao <ylmao@marvell.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This prepares *of_device_id.data becoming const. Without this change the
following warning would occur:

	arch/arm/mm/cache-l2x0.c: In function 'l2x0_of_init':
	arch/arm/mm/cache-l2x0.c:573:7: warning: assignment discards 'const' qualifier from pointer target type [enabled by default]

though.
Signed-off-by: NUwe Kleine-König <u.kleine-koenig@pengutronix.de>

The __free_from_pool() function was changed in
e9da6e99. Unfortunately, the test that
checks whether the provided (start,size) is within the DMA pool has
been improperly modified. It used to be:

  if (start < coherent_head.vm_start || end > coherent_head.vm_end)

Where coherent_head.vm_end was non-inclusive (i.e, it did not include
the first byte after the pool). The test has been changed to:

  if (start < pool->vaddr || start > pool->vaddr + pool->size)

So now pool->vaddr + pool->size is inclusive (i.e, it includes the
first byte after the pool), so the test should be >= instead of >.

This bug causes the following message when freeing the *first* DMA
coherent buffer that has been allocated, because its virtual address
is exactly equal to pool->vaddr + pool->size :

WARNING: at /home/thomas/projets/linux-2.6/arch/arm/mm/dma-mapping.c:463 __free_from_pool+0xa4/0xc0()
freeing wrong coherent size from pool
Signed-off-by: NThomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Lior Amsalem <alior@marvell.com>
Cc: Maen Suleiman <maen@marvell.com>
Cc: Tawfik Bayouk <tawfik@marvell.com>
Cc: Shadi Ammouri <shadi@marvell.com>
Cc: Eran Ben-Avi <benavi@marvell.com>
Cc: Yehuda Yitschak <yehuday@marvell.com>
Cc: Nadav Haklai <nadavh@marvell.com>
[m.szyprowski: rebased onto v3.6-rc5 and resolved conflict]
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Make use of the same atomic pool as DMA does, and skip a kernel page
mapping which can involve sleep'able operations at allocating a kernel
page table.
Signed-off-by: NHiroshi Doyu <hdoyu@nvidia.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Support atomic allocation in __iommu_get_pages().
Signed-off-by: NHiroshi Doyu <hdoyu@nvidia.com>
[moved __atomic_get_pages() under #ifdef CONFIG_ARM_DMA_USE_IOMMU
 to avoid unused fuction warning for no-IOMMU case]
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Check the given range("start", "size") is included in "atomic_pool" or not.
Signed-off-by: NHiroshi Doyu <hdoyu@nvidia.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

struct page **pages is necessary to align with non atomic path in
__iommu_get_pages(). atomic_pool() has the intialized **pages instead
of just *page.
Signed-off-by: NHiroshi Doyu <hdoyu@nvidia.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Print a loud warning when system runs out of memory from atomic DMA
coherent pool to let users notice the potential problem.
Reported-by: NAaro Koskinen <aaro.koskinen@iki.fi>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Some platforms might require to increase atomic coherent pool to make
sure that their device will be able to allocate all their buffers from
atomic context. This function can be also used to decrease atomic
coherent pool size if coherent allocations are not used for the given
sub-platform.
Suggested-by: NJosh Coombs <josh.coombs@gmail.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

With !HIGHMEM, sanity_check_meminfo checks for banks that completely or
partially overlap the vmalloc region. The test for partial overlap checks
__va(bank->start + bank->size) > vmalloc_min. This is not appropriate if
there is a non-linear translation between virtual and physical addresses,
as bank->start + bank->size is actually in the bank following the one being
interrogated.

In most cases, even when using SPARSEMEM, this is not problematic as the
subsequent bank will start at a higher va than the one in question. However
if the physical to virtual address conversion is not monotonic increasing,
the incorrect test could result in a bank not being truncated when it
should be.

This patch ensures we perform the va-pa conversion on memory from the
bank we are interested in, not the following one.
Reported-by: N??? (Steve) <zhanzhenbo@gmail.com>
Signed-off-by: NJonathan Austin <jonathan.austin@arm.com>
Acked-by: NNicolas Pitre <nico@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

The bfi instruction is not available on ARMv6, so instead use an and/orr
sequence in the contextidr_notifier. This gets rid of the assembler
error:

  Assembler messages:
  Error: selected processor does not support ARM mode `bfi r3,r2,#0,#8'
Reported-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Murali Nalajala reports a regression that ioremapping address zero
results in an oops dump:

Unable to handle kernel paging request at virtual address fa200000
pgd = d4f80000
[fa200000] *pgd=00000000
Internal error: Oops: 5 [#1] PREEMPT SMP ARM
Modules linked in:
CPU: 0    Tainted: G        W (3.4.0-g3b5f728-00009-g638207a #13)
PC is at msm_pm_config_rst_vector_before_pc+0x8/0x30
LR is at msm_pm_boot_config_before_pc+0x18/0x20
pc : [<c0078f84>]    lr : [<c007903c>]    psr: a0000093
sp : c0837ef0  ip : cfe00000  fp : 0000000d
r10: da7efc17  r9 : 225c4278  r8 : 00000006
r7 : 0003c000  r6 : c085c824  r5 : 00000001  r4 : fa101000
r3 : fa200000  r2 : c095080c  r1 : 002250fc  r0 : 00000000
Flags: NzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM Segment kernel
Control: 10c5387d  Table: 25180059  DAC: 00000015
[<c0078f84>] (msm_pm_config_rst_vector_before_pc+0x8/0x30) from [<c007903c>] (msm_pm_boot_config_before_pc+0x18/0x20)
[<c007903c>] (msm_pm_boot_config_before_pc+0x18/0x20) from [<c007a55c>] (msm_pm_power_collapse+0x410/0xb04)
[<c007a55c>] (msm_pm_power_collapse+0x410/0xb04) from [<c007b17c>] (arch_idle+0x294/0x3e0)
[<c007b17c>] (arch_idle+0x294/0x3e0) from [<c000eed8>] (default_idle+0x18/0x2c)
[<c000eed8>] (default_idle+0x18/0x2c) from [<c000f254>] (cpu_idle+0x90/0xe4)
[<c000f254>] (cpu_idle+0x90/0xe4) from [<c057231c>] (rest_init+0x88/0xa0)
[<c057231c>] (rest_init+0x88/0xa0) from [<c07ff890>] (start_kernel+0x3a8/0x40c)
Code: c0704256 e12fff1e e59f2020 e5923000 (e5930000)

This is caused by the 'reserved' entries which we insert (see
19b52abe - ARM: 7438/1: fill possible PMD empty section gaps)
which get matched for physical address zero.

Resolve this by marking these reserved entries with a different flag.

Cc: <stable@vger.kernel.org>
Tested-by: NMurali Nalajala <mnalajal@codeaurora.org>
Acked-by: NNicolas Pitre <nico@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Signed-off-by: NChao Xie <xiechao.mail@gmail.com>
Signed-off-by: NHaojian Zhuang <haojian.zhuang@gmail.com>

The extra feature may be used by SOCs are prefetch, burst8,
write buffer coalesce
Signed-off-by: NChao Xie <xiechao.mail@gmail.com>
Signed-off-by: NHaojian Zhuang <haojian.zhuang@gmail.com>

Signed-off-by: NChao Xie <xiechao.mail@gmail.com>
Signed-off-by: NHaojian Zhuang <haojian.zhuang@gmail.com>

init the variable "mode" to NULL to ensure the later NULL checking is
taking effect.
Signed-off-by: NChao Xie <xiechao.mail@gmail.com>
Signed-off-by: NHaojian Zhuang <haojian.zhuang@gmail.com>

Allow arm_memblock_steal() to remove memory from any RAM region,
including highmem areas.  This allows memory to be stolen from the
very top of declared memory, including highmem areas, rather than
our precious lowmem.
Acked-by: NSascha Hauer <s.hauer@pengutronix.de>
Acked-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Commit 5a783cbc ("ARM: 7478/1: errata: extend workaround for erratum
 #720789") added workarounds for erratum #720789 to the range TLB
invalidation functions with the observation that the erratum only
affects SMP platforms. However, when running an SMP_ON_UP kernel on a
uniprocessor platform we must take care to preserve the ASID as the
workaround is not required.

This patch ensures that we don't set the ASID to 0 when flushing the TLB
on such a system, preserving the original behaviour with the workaround
disabled.

Cc: <stable@vger.kernel.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Swap entries are encoding in ptes such that !pte_present(pte) and
pte_file(pte). The remaining bits of the descriptor are used to identify
the swapfile and offset within it to the swap entry.

When writing such a pte for a user virtual address, set_pte_at
unconditionally sets the nG bit, which (in the case of LPAE) will
corrupt the swapfile offset and lead to a BUG:

[  140.494067] swap_free: Unused swap offset entry 000763b4
[  140.509989] BUG: Bad page map in process rs:main Q:Reg  pte:0ec76800 pmd:8f92e003

This patch fixes the problem by only setting the nG bit for user
mappings that are actually present.

Cc: <stable@vger.kernel.org>
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Commit e9da6e99 (ARM: dma-mapping:
remove custom consistent dma region) changed the way atomic allocations
are handled. However, arm_dma_free() was not modified accordingly, and
as a result freeing of atomic allocations does not work correctly when
CMA is disabled. Memory is leaked and following WARNINGs are seen:

[   57.698911] ------------[ cut here ]------------
[   57.753518] WARNING: at arch/arm/mm/dma-mapping.c:263 arm_dma_free+0x88/0xe4()
[   57.811473] trying to free invalid coherent area: e0848000
[   57.867398] Modules linked in: sata_mv(-)
[   57.921373] [<c000d270>] (unwind_backtrace+0x0/0xf0) from [<c0015430>] (warn_slowpath_common+0x50/0x68)
[   58.033924] [<c0015430>] (warn_slowpath_common+0x50/0x68) from [<c00154dc>] (warn_slowpath_fmt+0x30/0x40)
[   58.152024] [<c00154dc>] (warn_slowpath_fmt+0x30/0x40) from [<c000dc18>] (arm_dma_free+0x88/0xe4)
[   58.219592] [<c000dc18>] (arm_dma_free+0x88/0xe4) from [<c008fa30>] (dma_pool_destroy+0x100/0x148)
[   58.345526] [<c008fa30>] (dma_pool_destroy+0x100/0x148) from [<c019a64c>] (release_nodes+0x144/0x218)
[   58.475782] [<c019a64c>] (release_nodes+0x144/0x218) from [<c0197e10>] (__device_release_driver+0x60/0xb8)
[   58.614260] [<c0197e10>] (__device_release_driver+0x60/0xb8) from [<c0198608>] (driver_detach+0xd8/0xec)
[   58.756527] [<c0198608>] (driver_detach+0xd8/0xec) from [<c0197c54>] (bus_remove_driver+0x7c/0xc4)
[   58.901648] [<c0197c54>] (bus_remove_driver+0x7c/0xc4) from [<c004bfac>] (sys_delete_module+0x19c/0x220)
[   59.051447] [<c004bfac>] (sys_delete_module+0x19c/0x220) from [<c0009140>] (ret_fast_syscall+0x0/0x2c)
[   59.207996] ---[ end trace 0745420412c0325a ]---
[   59.287110] ------------[ cut here ]------------
[   59.366324] WARNING: at arch/arm/mm/dma-mapping.c:263 arm_dma_free+0x88/0xe4()
[   59.450511] trying to free invalid coherent area: e0847000
[   59.534357] Modules linked in: sata_mv(-)
[   59.616785] [<c000d270>] (unwind_backtrace+0x0/0xf0) from [<c0015430>] (warn_slowpath_common+0x50/0x68)
[   59.790030] [<c0015430>] (warn_slowpath_common+0x50/0x68) from [<c00154dc>] (warn_slowpath_fmt+0x30/0x40)
[   59.972322] [<c00154dc>] (warn_slowpath_fmt+0x30/0x40) from [<c000dc18>] (arm_dma_free+0x88/0xe4)
[   60.070701] [<c000dc18>] (arm_dma_free+0x88/0xe4) from [<c008fa30>] (dma_pool_destroy+0x100/0x148)
[   60.256817] [<c008fa30>] (dma_pool_destroy+0x100/0x148) from [<c019a64c>] (release_nodes+0x144/0x218)
[   60.445201] [<c019a64c>] (release_nodes+0x144/0x218) from [<c0197e10>] (__device_release_driver+0x60/0xb8)
[   60.634148] [<c0197e10>] (__device_release_driver+0x60/0xb8) from [<c0198608>] (driver_detach+0xd8/0xec)
[   60.823623] [<c0198608>] (driver_detach+0xd8/0xec) from [<c0197c54>] (bus_remove_driver+0x7c/0xc4)
[   61.013268] [<c0197c54>] (bus_remove_driver+0x7c/0xc4) from [<c004bfac>] (sys_delete_module+0x19c/0x220)
[   61.203472] [<c004bfac>] (sys_delete_module+0x19c/0x220) from [<c0009140>] (ret_fast_syscall+0x0/0x2c)
[   61.393390] ---[ end trace 0745420412c0325b ]---

The patch fixes this.
Signed-off-by: NAaro Koskinen <aaro.koskinen@iki.fi>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

The alignment mask is calculated incorrectly. Fixing the calculation
makes strange hangs/lockups disappear during the boot with Amstrad E3
and 3.6-rc1 kernel.
Signed-off-by: NAaro Koskinen <aaro.koskinen@iki.fi>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Fix dma_contiguous_remap() so that it continues through all the
regions, even after encountering one that is outside lowmem.
Without this change, if you have two CMA regions, the first outside
lowmem and the seocnd inside lowmem, only the second one will get
set up in the MMU. Data written to that region then doesn't get
automatically flushed from the cache into memory.
Signed-off-by: NChris Brand <cbrand@broadcom.com>
[extended patch subject with 'fix' word]
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Commit cdf357f1 ("ARM: 6299/1: errata: TLBIASIDIS and TLBIMVAIS
operations can broadcast a faulty ASID") replaced by-ASID TLB flushing
operations with all-ASID variants to workaround A9 erratum #720789.

This patch extends the workaround to include the tlb_range operations,
which were overlooked by the original patch.

Cc: <stable@vger.kernel.org>
Tested-by: NSteve Capper <steve.capper@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch adds support for DMA_ATTR_SKIP_CPU_SYNC attribute for
dma_(un)map_(single,page,sg) functions family. It lets dma mapping clients
to create a mapping for the buffer for the given device without performing
a CPU cache synchronization. CPU cache synchronization can be skipped for
the buffers which it is known that they are already in 'device' domain (CPU
caches have been already synchronized or there are only coherent mappings
for the buffer). For advanced users only, please use it with care.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Reviewed-by: NKyungmin Park <kyungmin.park@samsung.com>

This patch adds support for dma_get_sgtable() function which is required
to let drivers to share the buffers allocated by DMA-mapping subsystem.

Generic implementation based on virt_to_page() is not suitable for ARM
dma-mapping subsystem.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Reviewed-by: NKyungmin Park <kyungmin.park@samsung.com>

This patch adds support for DMA_ATTR_NO_KERNEL_MAPPING attribute for
IOMMU allocations, what let drivers to save precious kernel virtual
address space for large buffers that are intended to be accessed only
from userspace.

This patch is heavily based on initial work kindly provided by Abhinav
Kochhar <abhinav.k@samsung.com>.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Reviewed-by: NKyungmin Park <kyungmin.park@samsung.com>

This patch fixes incorrect check in error path. When the allocation of
first page fails, the kernel ops appears due to accessing -1 element of
the pages array.
Reported-by: NSylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Add some sanity checks and forbid mmaping of buffers into vma areas larger
than allocated dma buffer.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>