The function cpu_resume currently lives in the .data section.
There's no reason for it to be there since we can use relative
instructions without a problem. Move a few cpu_resume data
structures out of the assembly file so the .data annotation
can be dropped completely and cpu_resume ends up in the read
only text section.
Reviewed-by: NKees Cook <keescook@chromium.org>
Reviewed-by: NMark Rutland <mark.rutland@arm.com>
Reviewed-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Tested-by: NMark Rutland <mark.rutland@arm.com>
Tested-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Tested-by: NKees Cook <keescook@chromium.org>
Acked-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NLaura Abbott <lauraa@codeaurora.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The hyp stub vectors are currently loaded using adr. This
instruction has a +/- 1MB range for the loading address. If
the alignment for sections is changed the address may be more
than 1MB away, resulting in reclocation errors. Switch to using
adrp for getting the address to ensure we aren't affected by the
location of the __hyp_stub_vectors.
Acked-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>
Tested-by: NMark Rutland <mark.rutland@arm.com>
Tested-by: NKees Cook <keescook@chromium.org>
Signed-off-by: NLaura Abbott <lauraa@codeaurora.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

handle_arch_irq isn't actually text, it's just a function pointer.
It doesn't need to be stored in the text section and doing so
causes problesm if we ever want to make the kernel text read only.
Declare handle_arch_irq as a proper function pointer stored in
the data section.
Reviewed-by: NKees Cook <keescook@chromium.org>
Reviewed-by: NMark Rutland <mark.rutland@arm.com>
Acked-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Tested-by: NMark Rutland <mark.rutland@arm.com>
Tested-by: NKees Cook <keescook@chromium.org>
Signed-off-by: NLaura Abbott <lauraa@codeaurora.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

While we currently expect self-hosted debug support to be identical
across CPUs, we don't currently sanity check this.

This patch adds logging of the ID_AA64DFR{0,1}_EL1 values and associated
sanity checking code.

It's not clear to me whether we need to check PMUVer, TraceVer, and
DebugVer, as we don't currently rely on these fields at all.
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

A missing newline in the WARN_TAINT_ONCE string results in ugly and
somewhat difficult to read output in the case of a sanity check failure,
as the next print does not appear on a new line:

  Unsupported CPU feature variation.Modules linked in:

This patch adds the missing newline, fixing the output formatting.
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

It seems that Cortex-A53 r0p4 added support for AIFSR and ADFSR, and
ID_MMFR0.AuxReg has been updated accordingly to report this fact. As
Cortex-A53 could be paired with CPUs which do not implement these
registers (e.g. all current revisions of Cortex-A57), this may trigger a
sanity check failure at boot.

The AuxReg value describes the availability of the ACTLR, AIFSR, and
ADFSR registers, which are only of use to 32-bit guest OSs, and have
IMPLEMENTATION DEFINED contents. Given the nature of these registers it
is likely that KVM will need to trap accesses regardless of whether the
CPUs are heterogeneous.

This patch masks out the ID_MMFR0.AuxReg value from the sanity checks,
preventing spurious warnings at boot time.
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Reported-by: NAndre Przywara <andre.przywara@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The only requirement the scheduler has on cluster IDs is that they must
be unique.  When enumerating the topology based on MPIDR information the
kernel currently generates cluster IDs by using the first level of
affinity above the core ID (either level one or two depending on if the
core has multiple threads) however the ARMv8 architecture allows for up
to three levels of affinity.  This means that an ARMv8 system may
contain cores which have MPIDRs identical other than affinity level
three which with current code will cause us to report multiple cores
with the same identification to the scheduler in violation of its
uniqueness requirement.

Ensure that we do not violate the scheduler requirements on systems that
uses all the affinity levels by incorporating both affinity levels two
and three into the cluser ID when the cores are not threaded.

While no currently known hardware uses multi-level clusters it is better
to program defensively, this will help ease bringup of systems that have
them and will ensure that things like distribution install media do not
need to be respun to replace kernels in order to deploy such systems.
In the worst case the system will work but perform suboptimally until a
kernel modified to handle the new topology better is installed, in the
best case this will be an adequate description of such topologies for
the scheduler to perform well.
Signed-off-by: NMark Brown <broonie@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Not all of the errata we have workarounds for apply necessarily to all
SoCs, so people compiling a kernel for one very specific SoC may not
need to patch the kernel.
Introduce a new submenu in the "Platform selection" menu to allow
people to turn off certain bugs if they are not affected. By default
all of them are enabled.
Normal users or distribution kernels shouldn't bother to deselect any
bugs here, since the alternatives framework will take care of
patching them in only if needed.
Signed-off-by: NAndre Przywara <andre.przywara@arm.com>
[will: moved kconfig menu under `Kernel Features']
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The ARM erratum 832075 applies to certain revisions of Cortex-A57,
one of the workarounds is to change device loads into using
load-aquire semantics.
This is achieved using the alternatives framework.
Signed-off-by: NAndre Przywara <andre.przywara@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The ARM errata 819472, 826319, 827319 and 824069 define the same
workaround for these hardware issues in certain Cortex-A53 parts.
Use the new alternatives framework and the CPU MIDR detection to
patch "cache clean" into "cache clean and invalidate" instructions if
an affected CPU is detected at runtime.
Signed-off-by: NAndre Przywara <andre.przywara@arm.com>
[will: add __maybe_unused to squash gcc warning]
Signed-off-by: NWill Deacon <will.deacon@arm.com>

After each CPU has been started, we iterate through a list of
CPU features or bugs to detect CPUs which need (or could benefit
from) kernel code patches.
For each feature/bug there is a function which checks if that
particular CPU is affected. We will later provide some more generic
functions for common things like testing for certain MIDR ranges.
We do this for every CPU to cover big.LITTLE systems properly as
well.
If a certain feature/bug has been detected, the capability bit will
be set, so that later the call to apply_alternatives() will trigger
the actual code patching.
Signed-off-by: NAndre Przywara <andre.przywara@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

With a blatant copy of some x86 bits we introduce the alternative
runtime patching "framework" to arm64.
This is quite basic for now and we only provide the functions we need
at this time.
This is connected to the newly introduced feature bits.
Signed-off-by: NAndre Przywara <andre.przywara@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

For taking note if at least one CPU in the system needs a bug
workaround or would benefit from a code optimization, we create a new
bitmap to hold (artificial) feature bits.
Since elf_hwcap is part of the userland ABI, we keep it alone and
introduce a new data structure for that (along with some accessors).
Signed-off-by: NAndre Przywara <andre.przywara@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

update_insn_emulation_mode() returns 0 on success, so we should be
treating any non-zero values as failure, rather than the other way
around. Otherwise, writes to the sysctl file controlling the emulation
are ignored and immediately rolled back.
Reported-by: NGene Hackmann <ghackmann@google.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Translation faults that occur due to the input address being outside
of the address range mapped by the relevant base register are reported
as level 0 faults in ESR.DFSC.

If the faulting access cannot be resolved by the kernel (e.g. because
it is not mapped by a vma), then we report "input address range fault"
on the console. This was fine until we added support for 48-bit VAs,
which actually place PGDs at level 0 and can trigger faults for invalid
addresses that are within the range of the page tables.

This patch changes the string to report "level 0 translation fault",
which is far less confusing.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Having the instruction emulation submenu underneath "platform selection"
is a great way to hide options we don't want people to use, but somewhat
confusing when you stumble across it there.

Move the menuconfig option underneath "kernel features", where it makes
a bit more sense.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Introduce an event to trace the usage of emulated instructions. The
trace event is intended to help identify and encourage the migration
of legacy software using the emulation features.

Use this event to trace usage of swp and CP15 barrier emulation.
Acked-by: NSteven Rostedt <rostedt@goodmis.org>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NPunit Agrawal <punit.agrawal@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The CP15 barrier instructions (CP15ISB, CP15DSB and CP15DMB) are
deprecated in the ARMv7 architecture, superseded by ISB, DSB and DMB
instructions respectively. Some implementations may provide the
ability to disable the CP15 barriers by disabling the CP15BEN bit in
SCTLR_EL1. If not enabled, the encodings for these instructions become
undefined.

To support legacy software using these instructions, this patch
register hooks to -
* emulate CP15 barriers and warn the user about their use
* toggle CP15BEN in SCTLR_EL1
Signed-off-by: NPunit Agrawal <punit.agrawal@arm.com>
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The SWP instruction was deprecated in the ARMv6 architecture. The
ARMv7 multiprocessing extensions mandate that SWP/SWPB instructions
are treated as undefined from reset, with the ability to enable them
through the System Control Register SW bit. With ARMv8, the option to
enable these instructions through System Control Register was dropped
as well.

To support legacy applications using these instructions, port the
emulation of the SWP and SWPB instructions from the arm port to arm64.
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NPunit Agrawal <punit.agrawal@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Typically, providing support for legacy instructions requires
emulating the behaviour of instructions whose encodings have become
undefined. If the instructions haven't been removed from the
architecture, there maybe an option in the implementation to turn
on/off the support for these instructions.

Create common infrastructure to support legacy instruction
emulation. In addition to emulation, also provide an option to support
hardware execution when supported. The default execution mode (one of
undef, emulate, hw exeuction) is dependent on the state of the
instruction (deprecated or obsolete) in the architecture and
can specified at the time of registering the instruction handlers. The
runtime state of the emulation can be controlled by writing to
individual nodes in sysctl. The expected default behaviour is
documented as part of this patch.
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NPunit Agrawal <punit.agrawal@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Port support for AArch32 instruction condition code checking from arm
to arm64.
Signed-off-by: NPunit Agrawal <punit.agrawal@arm.com>
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Add support to register hooks for undefined instructions. The handlers
will be called when the undefined instruction and the processor state
(as contained in pstate) match criteria used at registration.
Signed-off-by: NPunit Agrawal <punit.agrawal@arm.com>
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The generic this_cpu operations disable interrupts to ensure that the
requested operation is protected from pre-emption. For arm64, this is
overkill and can hurt throughput and latency.

This patch provides arm64 specific implementations for the this_cpu
operations. Rather than disable interrupts, we use the exclusive
monitor or atomic operations as appropriate.

The following operations are implemented: add, add_return, and, or,
read, write, xchg. We also wire up a cmpxchg implementation from
cmpxchg.h.

Testing was performed using the percpu_test module and hackbench on a
Juno board running 3.18-rc4.
Signed-off-by: NSteve Capper <steve.capper@linaro.org>
Reviewed-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

We currently allocate different levels of page tables with a variety of
differing flags, and the PGALLOC_GFP flags, intended for use when
allocating any level of page table, are only used for ptes in
pte_alloc_one. On x86, PGALLOC_GFP is used for all page table
allocations.

Currently the major differences are:

* __GFP_NOTRACK -- Needed to ensure page tables are always accessible in
  the presence of kmemcheck to prevent recursive faults. Currently
  kmemcheck cannot be selected for arm64.

* __GFP_REPEAT -- Causes the allocator to try to reclaim pages and retry
  upon a failure to allocate.

* __GFP_ZERO -- Sometimes passed explicitly, sometimes zalloc variants
  are used.

While we've no encountered issues so far, it would be preferable to be
consistent. This patch ensures all levels of table are allocated in the
same manner, with PGALLOC_GFP.

Cc: Steve Capper <steve.capper@arm.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Since commit 8a0a9bd4 ('random: make get_random_int() more
random'), get_random_int() returns a random value for each call,
so comment and hack introduced in mmap_rnd() as part of commit
1d18c47c ('arm64: MMU fault handling and page table management')
are incorrects.

Commit 1d18c47c seems to use the same hack introduced by
commit a5adc91a ('powerpc: Ensure random space between stack
and mmaps'), latter copied in commit 5a0efea0 ('sparc64: Sharpen
address space randomization calculations.').

But both architectures were cleaned up as part of commit
fa8cbaaf ('powerpc+sparc64/mm: Remove hack in mmap randomize
layout') as hack is no more needed since commit 8a0a9bd4.

So the present patch removes the comment and the hack around
get_random_int() on AArch64's mmap_rnd().

Cc: David S. Miller <davem@davemloft.net>
Cc: Anton Blanchard <anton@samba.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: NWill Deacon <will.deacon@arm.com>
Acked-by: NDan McGee <dpmcgee@gmail.com>
Signed-off-by: NYann Droneaud <ydroneaud@opteya.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Commit a469abd0 (ARM: elf: add new hwcap for identifying atomic
ldrd/strd instructions) introduces HWCAP_ELF for 32-bit ARM
applications. As LPAE is always present on arm64, report the
corresponding compat HWCAP to user space.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Cc: <stable@vger.kernel.org> # 3.11+
Signed-off-by: NWill Deacon <will.deacon@arm.com>

On architectures with hardware broadcasting of TLB invalidation messages
, it makes sense to reduce the range of the mmu_gather structure when
unmapping page ranges based on the dirty address information passed to
tlb_remove_tlb_entry.

arm64 already does this by directly manipulating the start/end fields
of the gather structure, but this confuses the generic code which
does not expect these fields to change and can end up calculating
invalid, negative ranges when forcing a flush in zap_pte_range.

This patch moves the minimal range calculation out of the arm64 code
and into the generic implementation, simplifying zap_pte_range in the
process (which no longer needs to care about start/end, since they will
point to the appropriate ranges already). With the range being tracked
by core code, the need_flush flag is dropped in favour of checking that
the end of the range has actually been set.

Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Michal Simek <monstr@monstr.eu>
Acked-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The push/pop instructions can be suboptimal when saving/restoring large
amounts of data to/from the stack, for example on entry/exit from the
kernel. This is because:

  (1) They act on descending addresses (i.e. the newly decremented sp),
      which may defeat some hardware prefetchers

  (2) They introduce an implicit dependency between each instruction, as
      the sp has to be updated in order to resolve the address of the
      next access.

This patch removes the push/pop instructions from our kernel entry/exit
macros in favour of ldp/stp plus offset.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Using an explicit adr instruction to set the link register to point at
ret_fast_syscall/ret_to_user can defeat branch and return stack predictors.

Instead, use the standard calling instructions (bl, blr) and have an
unconditional branch as the following instruction.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Commit d7a49086 (arm64: cpuinfo: print info for all CPUs)
attempted to clean up /proc/cpuinfo, but due to concerns regarding
further changes was reverted in commit 5e39977e (Revert "arm64:
cpuinfo: print info for all CPUs").

There are two major issues with the arm64 /proc/cpuinfo format
currently:

* The "Features" line describes (only) the 64-bit hwcaps, which is
  problematic for some 32-bit applications which attempt to parse it. As
  the same names are used for analogous ISA features (e.g. aes) despite
  these generally being architecturally unrelated, it is not possible to
  simply append the 64-bit and 32-bit hwcaps in a manner that might not
  be misleading to some applications.

  Various potential solutions have appeared in vendor kernels. Typically
  the format of the Features line varies depending on whether the task
  is 32-bit.

* Information is only printed regarding a single CPU. This does not
  match the ARM format, and does not provide sufficient information in
  big.LITTLE systems where CPUs are heterogeneous. The CPU information
  printed is queried from the current CPU's registers, which is racy
  w.r.t. cross-cpu migration.

This patch attempts to solve these issues. The following changes are
made:

* When a task with a LINUX32 personality attempts to read /proc/cpuinfo,
  the "Features" line contains the decoded 32-bit hwcaps, as with the
  arm port. Otherwise, the decoded 64-bit hwcaps are shown. This aligns
  with the behaviour of COMPAT_UTS_MACHINE and COMPAT_ELF_PLATFORM. In
  the absense of compat support, the Features line is empty.

  The set of hwcaps injected into a task's auxval are unaffected.

* Properties are printed per-cpu, as with the ARM port. The per-cpu
  information is queried from pre-recorded cpu information (as used by
  the sanity checks).

* As with the previous attempt at fixing up /proc/cpuinfo, the hardware
  field is removed. The only users so far are 32-bit applications tied
  to particular boards, so no portable applications should be affected,
  and this should prevent future tying to particular boards.

The following differences remain:

* No model_name is printed, as this cannot be queried from the hardware
  and cannot be provided in a stable fashion. Use of the CPU
  {implementor,variant,part,revision} fields is sufficient to identify a
  CPU and is portable across arm and arm64.

* The following system-wide properties are not provided, as they are not
  possible to provide generally. Programs relying on these are already
  tied to particular (32-bit only) boards:
  - Hardware
  - Revision
  - Serial

No software has yet been identified for which these remaining
differences are problematic.

Cc: Greg Hackmann <ghackmann@google.com>
Cc: Ian Campbell <ijc@hellion.org.uk>
Cc: Serban Constantinescu <serban.constantinescu@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: cross-distro@lists.linaro.org
Cc: linux-api@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Replace two instances of 'ldr xN, =(constant)' in the world switch
hot path with 'mov' instructions.
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>
Acked-by: NChristoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Replace ldr xN, =<symbol> with adrp/add or adrp/ldr [as appropriate]
in the implementation of _mcount(), which may be called very often.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

In certain debugging scenarios it's useful to know the physical ID (i.e.
the MPIDR_EL1.Aff* fields) of the boot CPU, but we don't currently log
this as we do for 32-bit ARM kernels.

This patch makes the kernel log the physical ID of the boot CPU early in
the boot process. The CPU logical map initialisation is folded in to
smp_setup_processor_id (which contrary to its name is also called by UP
kernels). This is called before setup_arch, so should not adversely
affect existing cpu_logical_map users.
Acked-by: NSudeep Holla <sudeep.holla@arm.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Acked-by: NLorenzo Pieralisis <lorenzo.pieralisi@arm.com>
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This patch implements the AES key schedule generation using ARMv8
Crypto Instructions. It replaces the table based C implementation
in aes_generic.ko, which means we can drop the dependency on that
module.
Tested-by: NSteve Capper <steve.capper@linaro.org>
Acked-by: NSteve Capper <steve.capper@linaro.org>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Some of the macros defined in kvm_arm.h are useful in assembly files, but are
not compatible with the assembler.  Change any C language integer constant
definitions using appended U, UL, or ULL to the UL() preprocessor macro.  Also,
add a preprocessor include of the asm/memory.h file which defines the UL()
macro.

Fixes build errors like these when using kvm_arm.h in assembly
source files:

  Error: unexpected characters following instruction at operand 3 -- `and x0,x1,#((1U<<25)-1)'
Acked-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NGeoff Levand <geoff@infradead.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This installs the machine name as recorded by setup_machine_fdt()
as dump stack arch description. This results in the string to be
included in call stack dumps, as is shown here:

  ...
  Bad mode in Synchronous Abort handler detected, code 0x84000005
  CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.18.0-rc2+ #548
> Hardware name: linux,dummy-virt (DT)
  task: ffffffc07c870000 ti: ffffffc07c878000 task.ti: ffffffc07c878000
  PC is at 0x0
  ...

Note that systems that support DMI/SMBIOS may override this later.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The arm64 architecture has the ability to exclusively load and store
a pair of registers from an address (ldxp/stxp). Also the SLUB can take
advantage of a cmpxchg_double implementation to avoid taking some
locks.

This patch provides an implementation of cmpxchg_double for 64-bit
pairs, and activates the logic required for the SLUB to use these
functions (HAVE_ALIGNED_STRUCT_PAGE and HAVE_CMPXCHG_DOUBLE).

Also definitions of this_cpu_cmpxchg_8 and this_cpu_cmpxchg_double_8
are wired up to cmpxchg_local and cmpxchg_double_local (rather than the
stock implementations that perform non-atomic operations with
interrupts disabled) as they are used by the SLUB.

On a Juno platform running on only the A57s I get quite a noticeable
performance improvement with 5 runs of hackbench on v3.17:

         Baseline | With Patch
 -----------------+-----------
 Mean    119.2312 | 106.1782
 StdDev    0.4919 |   0.4494

(times taken to complete `./hackbench 100 process 1000', in seconds)
Signed-off-by: NSteve Capper <steve.capper@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Optimize memcpy_{from,to}io() and memset_io() by transferring in 64 bit
as much as possible with minimized barrier usage.  This simplest
optimization brings faster throughput compare to current byte-by-byte read
and write with barrier in the loop.  Code's skeleton is taken from the
powerpc.

Link: http://lkml.kernel.org/p/20141020133304.GH23751@e104818-lin.cambridge.arm.comReviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Reviewed-by: NTrilok Soni <tsoni@codeaurora.org>
Signed-off-by: NJoonwoo Park <joonwoop@codeaurora.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Use phys_addr_t for physical address in alloc_init_pud. Although
phys_addr_t and unsigned long are 64 bit in arm64, it is better
to use phys_addr_t to describe physical addresses.
Signed-off-by: NMin-Hua Chen <orca.chen@gmail.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Pull UEFI updates from Ard Biesheuvel:

  - fixes for compliance with PE/COFF and UEFI specs
  - added support for SMBIOS, including upcoming version 3.0
  - cleanups and diagnostic output improvements
Signed-off-by: NWill Deacon <will.deacon@arm.com>