We don't want GCC optimising our memset_io(), memcpy_fromio() or
memcpy_toio() variants, so we must not call one of the standard
functions.  Provide a separate name for our assembly memcpy() and
memset() functions, and use that instead, thereby bypassing GCC's
ability to optimise these operations.

GCCs optimisation may introduce unaligned accesses which are invalid
for device mappings.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert the ioremap*() preprocessor macros to real functions, moving
them out of line.  This allows us to kill off __arm_ioremap(), and
__arm_iounmap() helpers, and remove __arm_ioremap_pfn_caller() from
global view.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

ioremap_wt() was added by aliasing it to ioremap_nocache(), which is a
device mapping.  Device mappings do not allow unaligned accesses, but
it appears that GCC is able to inline its own memcpy() implementation
which may use such accesses.  The only user of this is pmem, which
uses memcpy() on the region.

Therefore, this is unsafe.  We must implement ioremap_wt() correctly
for ARM, or not at all.

This patch adds a more correct implementation by re-using ioremap_wc()
to provide a normal-memory non-cacheable mapping.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Add documentation of the ARM specific behaviour of the mappings setup by
the ioremap() series of macros.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Add ioremap_wt() to all arch-specific asm/io.h headers which
define ioremap_wc() locally. These headers do not include
<asm-generic/iomap.h>. Some of them include <asm-generic/io.h>,
but ioremap_wt() is defined for consistency since they define
all ioremap_xxx locally.

In all architectures without Write-Through support, ioremap_wt()
is defined indentical to ioremap_nocache().

frv and m68k already have ioremap_writethrough(). On those we
add ioremap_wt() indetical to ioremap_writethrough() and defines
ARCH_HAS_IOREMAP_WT in both architectures.

The ioremap_wt() interface is exported to drivers.
Signed-off-by: NToshi Kani <toshi.kani@hp.com>
Signed-off-by: NBorislav Petkov <bp@suse.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Elliott@hp.com
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Luis R. Rodriguez <mcgrof@suse.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: arnd@arndb.de
Cc: hch@lst.de
Cc: hmh@hmh.eng.br
Cc: jgross@suse.com
Cc: konrad.wilk@oracle.com
Cc: linux-mm <linux-mm@kvack.org>
Cc: linux-nvdimm@lists.01.org
Cc: stefan.bader@canonical.com
Cc: yigal@plexistor.com
Link: http://lkml.kernel.org/r/1433436928-31903-9-git-send-email-bp@alien8.deSigned-off-by: NIngo Molnar <mingo@kernel.org>

If we are building for a LE platform, and we haven't overriden the
MMIO ops, then we can optimize the mem*io operations using the
standard string functions.
Acked-by: NNicolas Pitre <nico@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

commit 195bbcac ("ARM: 7500/1: io: avoid writeback addressing
modes for __raw_ accessors") disables writeback addressing modes for
raw i/o. However, the "+Q" output constraint forces the compiler to
disable load hoist optimizations (because the output constraint informs the
compiler of memory stores which the compiler assumes may alias other memory).

Since the relaxed accessors only guarantee ordering wrt i/o accesses to the
same device and not to main memory, there's never a possibility of an accessor
invalidating a hoisted load (because only non-i/o loads would have been hoisted).

The effect is especially noticable with complex address inputs in loops.
For example, the following code:

    #include <linux/kernel.h>
    #include <linux/io.h>

    static const int *remap;

    void wr_loop(void __iomem *base, int c, int val)
    {
            int i;

            for (i = 0; i < c; i++)
                    writew_relaxed(val, base + remap[c >> 2]);
    }

generates

           current master             |             this patch
 0: e3510000    cmp     r1, #0        |  0: e3510000    cmp     r1, #0
 4: d12fff1e    bxle    lr            |  4: d12fff1e    bxle    lr
 8: e3003000    movw    r3, #0        |  8: e3c1c003    bic     ip, r1, #3
 c: e3403000    movt    r3, #0        |  c: e6ff2072    uxth    r2, r2
10: e92d4010    push    {r4, lr}      | 10: e3a03000    mov     r3, #0
14: e6ff2072    uxth    r2, r2        | 14: e59cc000    ldr     ip, [ip]
18: e3c14003    bic     r4, r1, #3    | 18: e080000c    add     r0, r0, ip
1c: e593e000    ldr     lr, [r3]      |
20: e3a03000    mov     r3, #0        | 1c: e1c020b0    strh    r2, [r0]
                                      | 20: e2833001    add     r3, r3, #1
24: e79ec004    ldr     ip, [lr, r4]  | 24: e1530001    cmp     r3, r1
28: e080c00c    add     ip, r0, ip    | 28: 1afffffb    bne     1c
2c: e1cc20b0    strh    r2, [ip]      | 2c: e12fff1e    bx      lr
30: e2833001    add     r3, r3, #1    |
34: e1530001    cmp     r3, r1        |
38: 1afffff9    bne     24            |
                                      |
3c: e8bd8010    pop     {r4, pc}      |
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NPeter Hurley <peter@hurleysoftware.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Include the generic I/O header file so that duplicate implementations
can be removed. This will also help to establish consistency across more
architectures regarding which accessors they support.
Signed-off-by: NThierry Reding <treding@nvidia.com>

This is needed for calls into OF code that parses PCI ranges.  It signals
support for memory mapped PCI I/O accesses that are described by device
trees.
Signed-off-by: NLiviu Dudau <Liviu.Dudau@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
CC: Russell King <linux@arm.linux.org.uk>
CC: Rob Herring <robh+dt@kernel.org>

Due to a design incompatibility between the PCIe Marvell controller
and the Cortex-A9, stressing PCIe devices with a lot of traffic
quickly causes a deadlock.

One part of the workaround for this is to have all PCIe regions mapped
as strongly-ordered (MT_UNCACHED) instead of the default
MT_DEVICE. While the arch_ioremap_caller() mechanism allows
sub-architecture code to override ioremap(), used to map PCIe memory
regions, there isn't such a mechanism to override the behavior of
pci_ioremap_io().

This commit adds the arch_pci_ioremap_mem_type variable, initialized
to MT_DEVICE by default, and that sub-architecture code can
override. We have chosen to expose a single variable rather than
offering the possibility of overriding the entire pci_ioremap_io(),
because implementing pci_ioremap_io() requires calling functions
(get_mem_type()) that are private to the arch/arm/mm/ code.
Signed-off-by: NThomas Petazzoni <thomas.petazzoni@free-electrons.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Some SoC have MMIO regions that are shared across orthogonal
subsystems. This commit implements a possible solution for the
thread-safe access of such regions through a spinlock-protected API.

Concurrent access is protected with a single spinlock for the
entire MMIO address space. While this protects shared-registers,
it also serializes access to unrelated/unshared registers.

We add relaxed and non-relaxed variants, by using writel_relaxed and writel,
respectively. The rationale for this is that some users may not require
register write completion but only thread-safe access to a register.
Tested-by: NSebastian Hesselbarth <sebastian.hesselbarth@gmail.com>
Acked-by: NJason Cooper <jason@lakedaemon.net>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NEzequiel Garcia <ezequiel.garcia@free-electrons.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

ioremap_cache is more aligned with other architectures.
There are only 2 users of this in the kernel: pxa2xx-flash and Xen.

This fixes Xen build failures on arm64:

drivers/tty/hvc/hvc_xen.c:233:2: error: implicit declaration of function 'ioremap_cached' [-Werror=implicit-function-declaration]
drivers/xen/grant-table.c:1174:3: error: implicit declaration of function 'ioremap_cached' [-Werror=implicit-function-declaration]
drivers/xen/xenbus/xenbus_probe.c:778:4: error: implicit declaration of function 'ioremap_cached' [-Werror=implicit-function-declaration]
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Signed-off-by: NStefano Stabellini <stefano.stabellini@eu.citrix.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

ioremap_cache is more aligned with other architectures. There are only
2 users of this in the kernel: pxa2xx-flash and Xen.

This fixes Xen build failures on arm64 caused by commit c04e8e2f (arm64:
allow ioremap_cache() to use existing RAM mappings)

drivers/tty/hvc/hvc_xen.c:233:2: error: implicit declaration of function 'ioremap_cached' [-Werror=implicit-function-declaration]
drivers/xen/grant-table.c:1174:3: error: implicit declaration of function 'ioremap_cached' [-Werror=implicit-function-declaration]
drivers/xen/xenbus/xenbus_probe.c:778:4: error: implicit declaration of function 'ioremap_cached' [-Werror=implicit-function-declaration]
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Signed-off-by: NStefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>

This is similar to what it is done on X86: biovecs are prevented from merging
otherwise every dma requests would be forced to bounce on the swiotlb buffer.
Signed-off-by: NStefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>


Changes in v7:
- remove the extra autotranslate check in biomerge.c.

Several of the ioremap functions use unsigned long in places
resulting in truncation if physical addresses greater than
4G are passed in. Change the types of the functions and the
callers accordingly.

Cc: Krzysztof Halasa <khc@pm.waw.pl>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Stephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NLaura Abbott <lauraa@codeaurora.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Using the 'o' memory constraint in inline assembly can result in GCC
generating invalid immediate offsets for memory access instructions with
reduced addressing capabilities (i.e. smaller than 12-bit immediate
offsets):

  http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54983

As there is no constraint to specify the exact addressing mode we need,
fallback to using 'Q' exclusively for halfword I/O accesses. This may
emit an additional add instruction (using an extra register) in order
to construct the address but it will always be accepted by GAS.
Reported-by: NBastian Hecht <hechtb@googlemail.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch fixes the /dev/mem driver to use phys_addr_t for physical
addresses.  This is required on PAE systems, especially those that run
entirely out of >4G physical memory space.
Signed-off-by: NCyril Chemparathy <cyril@ti.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Data aborts taken to hyp mode do not provide a valid instruction
syndrome field in the HSR if the faulting instruction is a memory
access using a writeback addressing mode.

For hypervisors emulating MMIO accesses to virtual peripherals, taking
such an exception requires disassembling the faulting instruction in
order to determine the behaviour of the access. Since this requires
manually walking the two stages of translation, the world must be
stopped to prevent races against page aging in the guest, where the
first-stage translation is invalidated after the hypervisor has
translated to an IPA and the physical page is reused for something else.

This patch avoids taking this heavy performance penalty when running
Linux as a guest by ensuring that our I/O accessors do not make use of
writeback addressing modes.

Cc: Marc Zyngier <marc.zyngier@arm.com>
Reviewed-by: NArnd Bergmann <arnd@arndb.de>
Reviewed-by: NNicolas Pitre <nico@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This adds a fixed virtual mapping for PCI i/o addresses. The mapping is
located at the last 2MB of vmalloc region (0xfee00000-0xff000000). 2MB
is used to align with PMD size, but IO_SPACE_LIMIT is 1MB. The space
is reserved after .map_io and can be mapped at any time later with
pci_ioremap_io. Platforms which need early i/o mapping (e.g. for vga
console) can call pci_map_io_early in their .map_io function.

This has changed completely from the 1st implementation which only
supported creating the static mapping at .map_io.
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Acked-by: NNicolas Pitre <nico@linaro.org>

out[bwl]() had a side effect that gcc read-back from the register after
writing its value.  This causes a problem for at least 3c589_cs, which
spits out lots of "adapter failure, FIFO diagnostic register 2011."
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

With commit c334bc15 (ARM: make mach/io.h include optional), PCMCIA was
broken. PCMCIA depends on __io() returning a valid i/o address, and most
ARM platforms require IO_SPACE_LIMIT be set to 0xffffffff for PCMCIA. This
needs a better fix with a fixed i/o address mapping, but for now we just
restore things to the previous behavior.

This fixes at91, omap1, pxa and sa11xx. pxa needs io.h if PCI is enabled,
but PCMCIA is not. sa11xx already has IO_SPACE_LIMIT set to 0xffffffff,
so it doesn't need an io.h.
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Joachim Eastwood <joachim.eastwood@jotron.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Andrew Victor <linux@maxim.org.za>
Cc: Nicolas Ferre <nicolas.ferre@atmel.com>
Cc: Jean-Christophe Plagniol-Villard <plagnioj@jcrosoft.com>
Tested-by: Paul Parsons <lost.distance@yahoo.com> (pxa270)
Acked-by: NTony Lindgren <tony@atomide.com>
Signed-off-by: NOlof Johansson <olof@lixom.net>

Disintegrate asm/system.h for ARM.
Signed-off-by: NDavid Howells <dhowells@redhat.com>
cc: Russell King <linux@arm.linux.org.uk>
cc: linux-arm-kernel@lists.infradead.org

Several platforms create IOMEM defines for casting to 'void __iomem *',
and other platforms are incorrectly using __io() macro for the same
purpose. This creates a common definition and removes all the platform
specific versions. Rather than try to make linux/io.h and asm/io.h
assembly safe, the assembly version of IOMEM is moved into
asm/assembler.h.
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Sekhar Nori <nsekhar@ti.com>
Cc: Kevin Hilman <khilman@ti.com>
Acked-by: NH Hartley Sweeten <hsweeten@visionengravers.com>
Cc: Ryan Mallon <rmallon@gmail.com>
Cc: Eric Miao <eric.y.miao@gmail.com>
Cc: Haojian Zhuang <haojian.zhuang@marvell.com>
Acked-by: NDavid Brown <davidb@codeaurora.org>
Cc: Daniel Walker <dwalker@fifo99.com>
Cc: Bryan Huntsman <bryanh@codeaurora.org>
Cc: Sascha Hauer <kernel@pengutronix.de>
Cc: Shawn Guo <shawn.guo@linaro.org>
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NPaul Walmsley <paul@pwsan.com>
Acked-by: NViresh Kumar <viresh.kumar@st.com>
Cc: Rajeev Kumar <rajeev-dlh.kumar@st.com>
Cc: Colin Cross <ccross@android.com>
Cc: Olof Johansson <olof@lixom.net>
Cc: Stephen Warren <swarren@nvidia.com>
Acked-by: NLinus Walleij <linus.walleij@linaro.org>
Acked-by: NArnd Bergmann <arnd@arndb.de>

__mem_pci is only used to enable readl/writel and friends. Just condition
this on readl being defined and remove all the __mem_pci defines.
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Lennert Buytenhek <kernel@wantstofly.org>
Cc: Imre Kaloz <kaloz@openwrt.org>
Cc: Krzysztof Halasa <khc@pm.waw.pl>
Cc: Nicolas Pitre <nico@fluxnic.net>
Cc: Ben Dooks <ben-linux@fluff.org>
Cc: Kukjin Kim <kgene.kim@samsung.com>
Cc: Colin Cross <ccross@android.com>
Cc: Olof Johansson <olof@lixom.net>
Cc: Stephen Warren <swarren@nvidia.com>

Add a kconfig option NEED_MACH_IO_H to conditionally include mach/io.h.

Basing this on CONFIG_PCI and CONFIG_ISA doesn't quite work. Most ISA
platforms don't need mach/io.h, but ebsa110 does. Most PCI platforms need
mach/io.h for now, but ks8695 doesn't which means i/o accesses are broken.
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Acked-by: NH Hartley Sweeten <hsweeten@visionengravers.com>
Acked-by: NNicolas Pitre <nico@linaro.org>

Now that all custom ioremap/iounmap users are converted to runtime hook,
remove the compile time defines.
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Acked-by: NNicolas Pitre <nico@linaro.org>

We have compile time over-ride of ioremap and iounmap, but an run-time
override is needed for multi-platform builds. This adds an extra function
pointer check, but ioremap is not peformance critical. The option for
compile time selection remains.

The caller variant is used here to provide correct caller information as
ARM can only support level 0 for __builtin_return_address.
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Reviewed-by: NArnd Bergmann <arnd@arndb.de>
Reviewed-by: NNicolas Pitre <nico@linaro.org>

arm copied pci_iomap from generic code, probably to avoid
pulling the rest of iomap.c in.  Since that's in
a separate file now, we can reuse the common implementation.
Signed-off-by: NMichael S. Tsirkin <mst@redhat.com>

This allows mapping external memory such as SRAM for use.

This is needed for some small chunks of code, such as reprogramming
SDRAM memory source clocks that can't be executed in SDRAM. Other
use cases include some PM related code.
Acked-by: NNicolas Pitre <nicolas.pitre@linaro.org>
Acked-by: NAndres Salomon <dilinger@queued.net>
Signed-off-by: NTony Lindgren <tony@atomide.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This resolves the following sparse warning from readl() and other macros,
which ends up embedding readl_relaxed() using the same variable.

arch/arm/mach-tegra/dma.c:169:8: warning: symbol '__v' shadows an earlier one
arch/arm/mach-tegra/dma.c:169:8: originally declared here
Signed-off-by: NOlof Johansson <olof@lixom.net>
Acked-by: NNicolas Pitre <nicolas.pitre@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

These functions are used in some PCI drivers with big-endian
MMIO space, and they are trivial to add here.
Signed-off-by: NArnd Bergmann <arnd@arndb.de>

There are numerous broken references to Documentation files (in other
Documentation files, in comments, etc.). These broken references are
caused by typo's in the references, and by renames or removals of the
Documentation files. Some broken references are simply odd.

Fix these broken references, sometimes by dropping the irrelevant text
they were part of.
Signed-off-by: NPaul Bolle <pebolle@tiscali.nl>
Signed-off-by: NJiri Kosina <jkosina@suse.cz>

Add a default IO_SPACE_LIMIT definition.  Explain the chosen value and
suggest why platforms would want to make it larger.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Ensure that the ISA/PCI IO space accessors are properly ordered on
ARMv6+ architectures.  These should always be ordered with respect to
all other accesses.

This also fixes __iormb() and __iowmb() not being visible to ioread/
iowrite if a platform defines its own MMIO accessors.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

We don't need to repeat the same definitions of the ioremap*(),
once in terms of __arch_ioremap() and again in terms of __arm_ioremap().
Instead, if the platform hasn't provided an __arch_ioremap, define
this to be __arm_ioremap, and only define the ioremap*() set using
__arch_ioremap.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Defining iounmap() with arguments prevents it from being used as a
function pointer, causing platforms to work around this.  Instead,
define it to be a simple macro.

Do the same for __arch_io(re|un)map too.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

There are very few legitimate use cases, if any, for directly accessing
system RAM through /dev/mem.  So let's mimic what they do on x86 and
forbid it when CONFIG_STRICT_DEVMEM is turned on.
Signed-off-by: NNicolas Pitre <nicolas.pitre@linaro.org>

The ioread/iowrite accessors also need barriers as they're used in
place of readl/writel et.al. in portable drivers.  Create __iormb()
and __iowmb() which are conditionally defined to be barriers dependent
on ARM_DMA_MEM_BUFFERABLE, and always use these macros in the accessors.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

When the coherent DMA buffers are mapped as Normal Non-cacheable
(ARM_DMA_MEM_BUFFERABLE enabled), buffer accesses are no longer ordered
with Device memory accesses causing failures in device drivers that do
not use the mandatory memory barriers before starting a DMA transfer.
LKML discussions led to the conclusion that such barriers have to be
added to the I/O accessors:

http://thread.gmane.org/gmane.linux.kernel/683509/focus=686153
http://thread.gmane.org/gmane.linux.ide/46414
http://thread.gmane.org/gmane.linux.kernel.cross-arch/5250

This patch introduces a wmb() barrier to the write*() I/O accessors to
handle the situations where Normal Non-cacheable writes are still in the
processor (or L2 cache controller) write buffer before a DMA transfer
command is issued. For the read*() accessors, a rmb() is introduced
after the I/O to avoid speculative loads where the driver polls for a
DMA transfer ready bit.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>