The patch adds support for THP (transparent huge pages) to LPAE
systems. When this feature is enabled, the kernel tries to map
anonymous pages as 2MB sections where possible.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
[steve.capper@linaro.org: symbolic constants used, value of
PMD_SECT_SPLITTING adjusted, tlbflush.h included in pgtable.h,
added PROT_NONE support.]
Signed-off-by: NSteve Capper <steve.capper@linaro.org>
Reviewed-by: NWill Deacon <will.deacon@arm.com>

Many ARMv7 cores have hardware page table walkers that can read the L1
cache. This is discoverable from the ID_MMFR3 register, although this
can be expensive to access from the low-level set_pte functions and is a
pain to cache, particularly with multi-cluster systems.

A useful observation is that the multi-processing extensions for ARMv7
require coherent table walks, meaning that we can make use of ALT_SMP
patching in proc-v7-* to patch away the cache flush safely for these
cores.
Reported-by: NAlbin Tonnerre <Albin.Tonnerre@arm.com>
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

On Cortex-A15 (r0p0..r3p2) the TLBI/DSB are not adequately shooting down
all use of the old entries. This patch implements the erratum workaround
which consists of:

1. Dummy TLBIMVAIS and DSB on the CPU doing the TLBI operation.
2. Send IPI to the CPUs that are running the same mm (and ASID) as the
   one being invalidated (or all the online CPUs for global pages).
3. CPU receiving the IPI executes a DMB and CLREX (part of the exception
   return code already).
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

We no longer support any ARMv3 platforms, so remove the old tlbflushing
code.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The ARM architecture requires explicit branch predictor maintenance
when updating an instruction stream for a given virtual address. In
reality, this isn't so much of a burden because the branch predictor
is flushed during the cache maintenance required to make the new
instructions visible to the I-side of the processor.

However, there are still some cases where explicit flushing is required,
so add a local_bp_flush_all operation to deal with this.
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch removes support for ARMv3 CPUs, which haven't worked properly
for quite some time (see the FIXME comment in arch/arm/mm/fault.c).  The
only V3 parts left is the cache model for ARMv3, which is needed for some
odd reason by ARM740T CPUs, and being able to build with -march=armv3,
which is required for the RiscPC platform due to its bus structure.
Acked-by: NWill Deacon <will.deacon@arm.com>
Acked-by: NJean-Christophe PLAGNIOL-VILLARD <plagnioj@jcrosoft.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

The compiler does not conditionalize the assembly instructions for
the tlb operations, which leads to sub-optimal code being generated
when building a kernel for multiple CPUs.

We can tweak things fairly simply as the code fragment below shows:

    17f8:       e3120001        tst     r2, #1  ; 0x1
...
    1800:       0a000000        beq     1808 <handle_pte_fault+0x194>
    1804:       ee061f10        mcr     15, 0, r1, cr6, cr0, {0}
    1808:       e3120004        tst     r2, #4  ; 0x4
    180c:       0a000000        beq     1814 <handle_pte_fault+0x1a0>
    1810:       ee081f36        mcr     15, 0, r1, cr8, cr6, {1}
becomes:
    17f0:       e3120001        tst     r2, #1  ; 0x1
    17f4:       1e063f10        mcrne   15, 0, r3, cr6, cr0, {0}
    17f8:       e3120004        tst     r2, #4  ; 0x4
    17fc:       1e083f36        mcrne   15, 0, r3, cr8, cr6, {1}

Overall, for Realview with V6 and V7 CPUs configured:

   text    data     bss     dec     hex filename
4153998  207340 5371036 9732374  948116 ../build/realview/vmlinux.before
4153366  207332 5371036 9731734  947e96 ../build/realview/vmlinux.after
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch defines the (pte|pmd)val_t as u32 and changes the page table
types to be based on these. The PMD bits are converted to the
corresponding type using the _AT macro.

The flush_pmd_entry/clean_pmd_entry argument was changed to (void *) to
allow them to be used with both PGD and PMD pointers and avoid code
duplication.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Kernel space needs very little in the way of BTC maintanence as most
mappings which are created and destroyed are non-executable, and so
could never enter the instruction stream.

The case which does warrant BTC maintanence is when a module is loaded.
This creates a new executable mapping, but at that point the pages have
not been initialized with code and data, so at that point they contain
unpredictable information.  Invalidating the BTC at this stage serves
little useful purpose.

Before we execute module code, we call flush_icache_range(), which deals
with the BTC maintanence requirements.  This ensures that we have a BTC
maintanence operation before we execute code via the newly created
mapping.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

There's no need to noMMU to put tlb_flush() in asm/tlbflush.h - it's
part of the tlb shootdown interface.  Move it to asm/tlb.h instead, as
per x86.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

UP systems do not implement all the instructions that SMP systems have,
so in order to boot a SMP kernel on a UP system, we need to rewrite
parts of the kernel.

Do this using an 'alternatives' scheme, where the kernel code and data
is modified prior to initialization to replace the SMP instructions,
thereby rendering the problematical code ineffectual.  We use the linker
to generate a list of 32-bit word locations and their replacement values,
and run through these replacements when we detect a UP system.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

On SMP systems, there is a small chance of a PTE becoming visible to a
different CPU before the current cache maintenance operations in
update_mmu_cache(). To avoid this, cache maintenance must be handled in
set_pte_at() (similar to IA-64 and PowerPC).

This patch provides a unified VIPT cache handling mechanism and
implements the __sync_icache_dcache() function for ARMv6 onwards
architectures. It is called from set_pte_at() and replaces the
update_mmu_cache(). The latter is still used on VIVT hardware where a
vm_area_struct is required.
Tested-by: NRabin Vincent <rabin.vincent@stericsson.com>
Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

There are places in Linux where writes to newly allocated page cache
pages happen without a subsequent call to flush_dcache_page() (several
PIO drivers including USB HCD). This patch changes the meaning of
PG_arch_1 to be PG_dcache_clean and always flush the D-cache for a newly
mapped page in update_mmu_cache().

The patch also sets the PG_arch_1 bit in the DMA cache maintenance
function to avoid additional cache flushing in update_mmu_cache().
Tested-by: NRabin Vincent <rabin.vincent@stericsson.com>
Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

On versions of the Cortex-A9 prior to r2p0, performing TLB invalidations by
ASID match can result in the incorrect ASID being broadcast to other CPUs.
As a consequence of this, the targetted TLB entries are not invalidated
across the system.

This workaround changes the TLB flushing routines to invalidate entries
regardless of the ASID.

Cc: <stable@kernel.org>
Tested-by: NRob Clark <rob@ti.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

The standard I-cache Invalidate All (ICIALLU) and Branch Predication
Invalidate All (BPIALL) operations are not automatically broadcast to
the other CPUs in an ARMv7 MP system. The patch adds the Inner Shareable
variants, ICIALLUIS and BPIALLIS, if ARMv7 and SMP.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

On VIVT ARM, when we have multiple shared mappings of the same file
in the same MM, we need to ensure that we have coherency across all
copies.  We do this via make_coherent() by making the pages
uncacheable.

This used to work fine, until we allowed highmem with highpte - we
now have a page table which is mapped as required, and is not available
for modification via update_mmu_cache().

Ralf Beache suggested getting rid of the PTE value passed to
update_mmu_cache():

  On MIPS update_mmu_cache() calls __update_tlb() which walks pagetables
  to construct a pointer to the pte again.  Passing a pte_t * is much
  more elegant.  Maybe we might even replace the pte argument with the
  pte_t?

Ben Herrenschmidt would also like the pte pointer for PowerPC:

  Passing the ptep in there is exactly what I want.  I want that
  -instead- of the PTE value, because I have issue on some ppc cases,
  for I$/D$ coherency, where set_pte_at() may decide to mask out the
  _PAGE_EXEC.

So, pass in the mapped page table pointer into update_mmu_cache(), and
remove the PTE value, updating all implementations and call sites to
suit.

Includes a fix from Stephen Rothwell:

  sparc: fix fallout from update_mmu_cache API change
Signed-off-by: NStephen Rothwell <sfr@canb.auug.org.au>
Acked-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch fixes the BUG: using smp_processor_id() in preemptible
Below is the stripped backtrace.

BUG: using smp_processor_id() in preemptible [00000000] code: init/1
caller is flush_tlb_mm+0x44/0x70
Backtrace:
[<c00225c4>] (dump_backtrace+0x0/0x110) from [<c01713a0>] (dump_stack+0x18/0x1c)
 r7:00000000 r6:c00234f0 r5:00000001 r4:c7828000
[<c0171388>] (dump_stack+0x0/0x1c) from [<c0135364>] (debug_smp_processor_id+0xc0/0xf0)
[<c01352a4>] (debug_smp_processor_id+0x0/0xf0) from [<c00234f0>] (flush_tlb_mm+0x44/0x70)
 r7:00000000 r6:c60b41a0 r5:c60b4154 r4:00000001
[<c00234ac>] (flush_tlb_mm+0x0/0x70) from [<c0039568>] (dup_mm+0x304/0x38c)
 r5:c1f09058 r4:00000000
[<c0039264>] (dup_mm+0x0/0x38c) from [<c0039de4>] (copy_process+0x7b8/0xeb0)
[<c003962c>] (copy_process+0x0/0xeb0) from [<c003a638>] (do_fork+0x15c/0x29c)
[<c003a4dc>] (do_fork+0x0/0x29c) from [<c0021df0>] (sys_clone+0x34/0x3c)
[<c0021dbc>] (sys_clone+0x0/0x3c) from [<c001efa0>] (ret_fast_syscall+0x0/0x2c)
Signed-off-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Makes code futureproof against the impending change to mm->cpu_vm_mask.

It's also a chance to use the new cpumask_ ops which take a pointer
(the older ones are deprecated, but there's no hurry for arch code).
Signed-off-by: NRusty Russell <rusty@rustcorp.com.au>

ARMv7 SMP hardware can handle the TLB maintenance operations
broadcasting in hardware so that the software can avoid the costly IPIs.
This patch adds the necessary checks (the MMFR3 CPUID register) to avoid
the broadcasting if already supported by the hardware.

(this patch is based on the work done by Tony Thompson @ ARM)
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Adds support for Faraday FA526 core. This core is used at least by:
Cortina Systems Gemini and Centroid family
Cavium Networks ECONA family
Grain Media GM8120
Pixelplus ImageARM
Prolific PL-1029
Faraday IP evaluation boards

v2:
- move TLB_BTB to separate patch
- update copyrights
Signed-off-by: NPaulius Zaleckas <paulius.zaleckas@teltonika.lt>

Signed-off-by: NPaulius Zaleckas <paulius.zaleckas@teltonika.lt>

Commit 2ccdd1e7 doesn't add
v7wbi_possible_flags and v7wbi_always_flags to possible_tlb_flags and
always_tlb_flags.  This causes the L2 cache flush in clean_pmd_entry()
(intended for Feroceon only) to execute on ARMv7, and the CPU hangs.

This patch is required for OMAP3 boards to boot.
Signed-off-by: NPaul Walmsley <paul@pwsan.com>
Acked-by: NLennert Buytenhek <buytenh@wantstofly.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Move platform independent header files to arch/arm/include/asm, leaving
those in asm/arch* and asm/plat* alone.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch adds support for the unified Feroceon L2 cache controller
as found in e.g. the Marvell Kirkwood and Marvell Discovery Duo
families of ARM SoCs.

Note that:

- Page table walks are outer uncacheable on Kirkwood and Discovery
  Duo, since the ARMv5 spec provides no way to indicate outer
  cacheability of page table walks (specifying it in TTBR[4:3] is
  an ARMv6+ feature).

  This requires adding L2 cache clean instructions to
  proc-feroceon.S (dcache_clean_area(), set_pte()) as well as to
  tlbflush.h ({flush,clean}_pmd_entry()).  The latter case is handled
  by defining a new TLB type (TLB_FEROCEON) which is almost identical
  to the v4wbi one but provides a TLB_L2CLEAN_FR flag.

- The Feroceon L2 cache controller supports L2 range (i.e. 'clean L2
  range by MVA' and 'invalidate L2 range by MVA') operations, and this
  patch uses those range operations for all Linux outer cache
  operations, as they are faster than the regular per-line operations.

  L2 range operations are not interruptible on this hardware, which
  avoids potential livelock issues, but can be bad for interrupt
  latency, so there is a compile-time tunable (MAX_RANGE_SIZE) which
  allows you to select the maximum range size to operate on at once.
  (Valid range is between one cache line and one 4KiB page, and must
  be a multiple of the line size.)
Signed-off-by: NLennert Buytenhek <buytenh@marvell.com>

Nobody uses flush_tlb_pgtables anymore, this patch removes all remaining
traces of it from all archs.
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: <linux-arch@vger.kernel.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

We are currently using the ARMv6 operations but need to duplicate some
of the code because of the introduction of the new CPU barrier
instructions in ARMv7.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

First thing mm.h does is including sched.h solely for can_do_mlock() inline
function which has "current" dereference inside. By dealing with can_do_mlock()
mm.h can be detached from sched.h which is good. See below, why.

This patch
a) removes unconditional inclusion of sched.h from mm.h
b) makes can_do_mlock() normal function in mm/mlock.c
c) exports can_do_mlock() to not break compilation
d) adds sched.h inclusions back to files that were getting it indirectly.
e) adds less bloated headers to some files (asm/signal.h, jiffies.h) that were
   getting them indirectly

Net result is:
a) mm.h users would get less code to open, read, preprocess, parse, ... if
   they don't need sched.h
b) sched.h stops being dependency for significant number of files:
   on x86_64 allmodconfig touching sched.h results in recompile of 4083 files,
   after patch it's only 3744 (-8.3%).

Cross-compile tested on

	all arm defconfigs, all mips defconfigs, all powerpc defconfigs,
	alpha alpha-up
	arm
	i386 i386-up i386-defconfig i386-allnoconfig
	ia64 ia64-up
	m68k
	mips
	parisc parisc-up
	powerpc powerpc-up
	s390 s390-up
	sparc sparc-up
	sparc64 sparc64-up
	um-x86_64
	x86_64 x86_64-up x86_64-defconfig x86_64-allnoconfig

as well as my two usual configs.
Signed-off-by: NAlexey Dobriyan <adobriyan@gmail.com>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The architecture specification states that TLB operations are
guaranteed to be complete only after the execution of a DSB (Data
Synchronisation Barrier, former Data Write Barrier or Drain Write
Buffer). The branch target cache invalidation is also needed. The ISB
(Instruction Synchronisation Barrier, formerly Prefetch Flush) is
needed unless there will be a return from exception before the
corresponding mapping is used (i.e. user mappings).
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Patch from Daniel Jacobowitz

The ARM kernel has several uses of asm("foo%?").  %? is a GCC internal
modifier used to output conditional execution predicates.  However, no
version of GCC supports conditionalizing asm statements.  GCC 4.2 will
correctly expand %? to the empty string in user asms.  Earlier versions may
reuse the condition from the previous instruction.  In 'if (foo) asm
("bar%?");' this is somewhat likely to be right... but not reliable.

So, the only safe thing to do is to remove the uses of %?.  I believe
the tlbflush.h occurances were supposed to be removed before, based
on the comment about %? not working at the top of that file.

Old versions of GCC could omit branches around user asms if the asm didn't
mark the condition codes as clobbered.  This problem hasn't been seen on any
recent (3.x or 4.x) GCC, but it could theoretically happen.  So, where
%? was removed a cc clobber was added.
Signed-off-by: NDaniel Jacobowitz <dan@codesourcery.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Signed-off-by: NDavid Woodhouse <dwmw2@infradead.org>

s/CONFIG_MMY/CONFIG_MMU/
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch removes TLB related codes in nommu mode.
Signed-off-by: NHyok S. Choi <hyok.choi@samsung.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Patch from Catalin Marinas

Chapter B2.7.3 in the latest ARM ARM (with v6 information) states that
the completion of a TLB maintenance operation is only guaranteed by
the execution of a DSB (Data Syncronization Barrier, formerly Data
Write Barrier or Drain Write Buffer).

Note that a DSB is only needed in the flush_tlb_kernel_* functions
since the completion is guaranteed by a mode change (i.e. switching
back to user mode) for the flush_tlb_user_* functions.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Initial git repository build. I'm not bothering with the full history,
even though we have it. We can create a separate "historical" git
archive of that later if we want to, and in the meantime it's about
3.2GB when imported into git - space that would just make the early
git days unnecessarily complicated, when we don't have a lot of good
infrastructure for it.

Let it rip!