Whoops -- I must have just been being an idiot again.  Thanks to Segher
for finding the bug :).

CC: Segher Boessenkool <segher@kernel.crashing.org>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

The comments in the ASID allocator incorrectly hint at an MP-style idiom
using the asid_generation and the active_asids array. In fact, the
synchronisation is achieved using a combination of an xchg operation
and a spinlock, so update the comments and remove the pointless smp_wmb().

Cc: James Morse <james.morse@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Building the kernel with an LTO-enabled GCC spits out the following "const"
warning for the cpu_ops code:

  mm/percpu.c:2168:20: error: pcpu_fc_names causes a section type conflict
  with dt_supported_cpu_ops
  const char * const pcpu_fc_names[PCPU_FC_NR] __initconst = {
          ^
  arch/arm64/kernel/cpu_ops.c:34:37: note: ‘dt_supported_cpu_ops’ was declared here
  static const struct cpu_operations *dt_supported_cpu_ops[] __initconst = {

Fix it by adding missed const qualifiers.
Signed-off-by: NYury Norov <ynorov@caviumnetworks.com>
Reviewed-by: NNick Desaulniers <ndesaulniers@google.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

bus access read/write events are not supported in A73, based on the
Cortex-A73 TRM r0p2, section 11.9 Events (pages 11-457 to 11-460).

Fixes: 5561b6c5 "arm64: perf: add support for Cortex-A73"
Acked-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NXu YiPing <xuyiping@hisilicon.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The fpsimd_update_current_state() function is responsible for
loading the FPSIMD state from the user signal frame into the
current task during sigreturn.  When implementing support for SVE,
conditional code was added to this function in order to handle the
case where SVE state need to be loaded for the task and merged with
the FPSIMD data from the signal frame; however, the FPSIMD-only
case was unintentionally dropped.

As a result of this, sigreturn does not currently restore the
FPSIMD state of the task, except in the case where the system
supports SVE and the signal frame contains SVE state in addition to
FPSIMD state.

This patch fixes this bug by making the copy-in of the FPSIMD data
from the signal frame to thread_struct unconditional.

This remains a performance regression from v4.14, since the FPSIMD
state is now copied into thread_struct and then loaded back,
instead of _only_ being loaded into the CPU FPSIMD registers.
However, it is essential to call task_fpsimd_load() here anyway in
order to ensure that the SVE enable bit in CPACR_EL1 is set
correctly before returning to userspace.  This could use some
refactoring, but since sigreturn is not a fast path I have kept
this patch as a pure fix and left the refactoring for later.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Fixes: 8cd969d2 ("arm64/sve: Signal handling support")
Reported-by: NAlex Bennée <alex.bennee@linaro.org>
Tested-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NAlex Bennée <alex.bennee@linaro.org>
Signed-off-by: NDave Martin <Dave.Martin@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

pgd_cache is setup once while init stage and never changed after
that, so it is good candidate for __ro_after_init
Signed-off-by: NJinbum Park <jinb.park7@gmail.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

When building the arm64 kernel with both CONFIG_ARM64_MODULE_PLTS and
CONFIG_DYNAMIC_FTRACE enabled, the ftrace-mod.o object file is built
with the kernel and contains a trampoline that is linked into each
module, so that modules can be loaded far away from the kernel and
still reach the ftrace entry point in the core kernel with an ordinary
relative branch, as is emitted by the compiler instrumentation code
dynamic ftrace relies on.

In order to be able to build out of tree modules, this object file
needs to be included into the linux-headers or linux-devel packages,
which is undesirable, as it makes arm64 a special case (although a
precedent does exist for 32-bit PPC).

Given that the trampoline essentially consists of a PLT entry, let's
not bother with a source or object file for it, and simply patch it
in whenever the trampoline is being populated, using the existing
PLT support routines.

Cc: <stable@vger.kernel.org>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

To allow the ftrace trampoline code to reuse the PLT entry routines,
factor it out and move it into asm/module.h.

Cc: <stable@vger.kernel.org>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

We used to have some cmpxchg syscalls.  They're no longer there, so we
no longer need the include.

CC: Christoph Hellwig <hch@infradead.org>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

Despite RISC-V having a direct 'fence.i' instruction available to
userspace (which we can't trap!), that's not actually viable when
running on Linux because the kernel might schedule a process on another
hart.  There is no way for userspace to handle this without invoking the
kernel (as it doesn't know the thread->hart mappings), so we've defined
a RISC-V specific system call to flush the instruction cache.

This patch adds both a system call and a VDSO entry.  If possible, we'd
like to avoid having the system call be considered part of the
user-facing ABI and instead restrict that to the VDSO entry -- both just
in general to avoid having additional user-visible ABI to maintain, and
because we'd prefer that users just call the VDSO entry because there
might be a better way to do this in the future (ie, one that doesn't
require entering the kernel).
Signed-off-by: NAndrew Waterman <andrew@sifive.com>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

The RISC-V ISA allows for instruction caches that are not coherent WRT
stores, even on a single hart.  As a result, we need to explicitly flush
the instruction cache whenever marking a dirty page as executable in
order to preserve the correct system behavior.

Local instruction caches aren't that scary (our implementations actually
flush the cache, but RISC-V is defined to allow higher-performance
implementations to exist), but RISC-V defines no way to perform an
instruction cache shootdown.  When explicitly asked to do so we can
shoot down remote instruction caches via an IPI, but this is a bit on
the slow side.

Instead of requiring an IPI to all harts whenever marking a page as
executable, we simply flush the currently running harts.  In order to
maintain correct behavior, we additionally mark every other hart as
needing a deferred instruction cache which will be taken before anything
runs on it.
Signed-off-by: NAndrew Waterman <andrew@sifive.com>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

Fixes:

include/asm-generic/mm_hooks.h:20:11: warning: 'struct vm_area_struct' declared inside parameter list will not be visible outside of this definition or declaration
include/asm-generic/mm_hooks.h:19:38: warning: 'struct mm_struct' declared inside parameter list will not be visible outside of this definition or declaration
Signed-off-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

Signed-off-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

Fixes the following on allmodconfig build:

profile.c:(.text+0x3e4): undefined reference to `setup_profiling_timer'
Signed-off-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

These are the ones needed by current allmodconfig, so add them instead
of everything other architectures are exporting -- the rest can be
added on demand later if needed.
Signed-off-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

Needed by some modules (exported by other architectures).
Signed-off-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

include <linux/types.h> for __iomem definition. Also, add volatile to
iounmap() like other architectures have it to avoid "discarding
volatile" warnings from some drivers.

Finally, explicitly promote the base address for INB/OUTB functions to
avoid some old legacy drivers complaining about int-to-ptr promotions.
The drivers are unlikely to work but they're included in allmodconfig
so the warnings are noisy.

Fixes, among other warnings, these with allmodconfig:

../arch/riscv/include/asm/io.h:24:21: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token
 extern void __iomem *ioremap(phys_addr_t offset, unsigned long size);

sound/pci/echoaudio/echoaudio.c: In function 'snd_echo_free':
sound/pci/echoaudio/echoaudio.c:1879:10: warning: passing argument 1 of 'iounmap' discards 'volatile' qualifier from pointer target type [-Wdiscarded-qualifiers]
Signed-off-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

INT and SHORT are used by some drivers that pull in the include files,
so prefixing helps avoid namespace conflicts. Other constructs in the
same file already uses this.

Fixes, among others, these warnings with allmodconfig:

../sound/core/pcm_misc.c:43:0: warning: "INT" redefined
 #define INT __force int
Signed-off-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

Fixes this from allmodconfig:

drivers/tty/serial/earlycon.c:27:10: fatal error: asm/serial.h: No such file or directory
Signed-off-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

The 'access_permitted' helper is used in the gup-fast path and goes
beyond the simple _PAGE_RW check to also:

 - validate that the mapping is writable from a protection keys
   standpoint

 - validate that the pte has _PAGE_USER set since all fault paths where
   pmd_write is must be referencing user-memory.

Link: http://lkml.kernel.org/r/151043111049.2842.15241454964150083466.stgit@dwillia2-desk3.amr.corp.intel.comSigned-off-by: NDan Williams <dan.j.williams@intel.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: "Jérôme Glisse" <jglisse@redhat.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The 'access_permitted' helper is used in the gup-fast path and goes
beyond the simple _PAGE_RW check to also:

 - validate that the mapping is writable from a protection keys
   standpoint

 - validate that the pte has _PAGE_USER set since all fault paths where
   pud_write is must be referencing user-memory.

[dan.j.williams@intel.com: fix powerpc compile error]
  Link: http://lkml.kernel.org/r/151129127237.37405.16073414520854722485.stgit@dwillia2-desk3.amr.corp.intel.com
Link: http://lkml.kernel.org/r/151043110453.2842.2166049702068628177.stgit@dwillia2-desk3.amr.corp.intel.comSigned-off-by: NDan Williams <dan.j.williams@intel.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

In response to compile breakage introduced by a series that added the
pud_write helper to x86, Stephen notes:

    did you consider using the other paradigm:

    In arch include files:
    #define pud_write       pud_write
    static inline int pud_write(pud_t pud)
     .....

    Then in include/asm-generic/pgtable.h:

    #ifndef pud_write
    tatic inline int pud_write(pud_t pud)
    {
            ....
    }
    #endif

    If you had, then the powerpc code would have worked ... ;-) and many
    of the other interfaces in include/asm-generic/pgtable.h are
    protected that way ...

Given that some architecture already define pmd_write() as a macro, it's
a net reduction to drop the definition of __HAVE_ARCH_PMD_WRITE.

Link: http://lkml.kernel.org/r/151129126721.37405.13339850900081557813.stgit@dwillia2-desk3.amr.corp.intel.comSigned-off-by: NDan Williams <dan.j.williams@intel.com>
Suggested-by: NStephen Rothwell <sfr@canb.auug.org.au>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com>
Cc: Oliver OHalloran <oliveroh@au1.ibm.com>
Cc: Chris Metcalf <cmetcalf@mellanox.com>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Currently only get_user_pages_fast() can safely handle the writable gup
case due to its use of pud_access_permitted() to check whether the pud
entry is writable.  In the gup slow path pud_write() is used instead of
pud_access_permitted() and to date it has been unimplemented, just calls
BUG_ON().

    kernel BUG at ./include/linux/hugetlb.h:244!
    [..]
    RIP: 0010:follow_devmap_pud+0x482/0x490
    [..]
    Call Trace:
     follow_page_mask+0x28c/0x6e0
     __get_user_pages+0xe4/0x6c0
     get_user_pages_unlocked+0x130/0x1b0
     get_user_pages_fast+0x89/0xb0
     iov_iter_get_pages_alloc+0x114/0x4a0
     nfs_direct_read_schedule_iovec+0xd2/0x350
     ? nfs_start_io_direct+0x63/0x70
     nfs_file_direct_read+0x1e0/0x250
     nfs_file_read+0x90/0xc0

For now this just implements a simple check for the _PAGE_RW bit similar
to pmd_write.  However, this implies that the gup-slow-path check is
missing the extra checks that the gup-fast-path performs with
pud_access_permitted.  Later patches will align all checks to use the
'access_permitted' helper if the architecture provides it.

Note that the generic 'access_permitted' helper fallback is the simple
_PAGE_RW check on architectures that do not define the
'access_permitted' helper(s).

[dan.j.williams@intel.com: fix powerpc compile error]
  Link: http://lkml.kernel.org/r/151129126165.37405.16031785266675461397.stgit@dwillia2-desk3.amr.corp.intel.com
Link: http://lkml.kernel.org/r/151043109938.2842.14834662818213616199.stgit@dwillia2-desk3.amr.corp.intel.com
Fixes: a00cc7d9 ("mm, x86: add support for PUD-sized transparent hugepages")
Signed-off-by: NDan Williams <dan.j.williams@intel.com>
Reported-by: NStephen Rothwell <sfr@canb.auug.org.au>
Acked-by: Thomas Gleixner <tglx@linutronix.de>	[x86]
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: <stable@vger.kernel.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

If we don't put the NG4fls.o object into the same part of
the link as the generic sparc64 objects for fls() and __fls()
then the relocation in the branch we use for patching will
not fit.

Move NG4fls.o into lib-y to fix this problem.

Fixes: 46ad8d2d ("sparc64: Use sparc optimized fls and __fls for T4 and above")
Signed-off-by: NDavid S. Miller <davem@davemloft.net>
Reported-by: NAnatoly Pugachev <matorola@gmail.com>
Tested-by: NAnatoly Pugachev <matorola@gmail.com>

There is a fast-path of MMIO emulation inside hyp mode. The handling
of single-step is broadly the same as kvm_arm_handle_step_debug()
except we just setup ESR/HSR so handle_exit() does the correct thing
as we exit.

For the case of an emulated illegal access causing an SError we will
exit via the ARM_EXCEPTION_EL1_SERROR path in handle_exit(). We behave
as we would during a real SError and clear the DBG_SPSR_SS bit for the
emulated instruction.
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>
Reviewed-by: NChristoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Signed-off-by: NChristoffer Dall <christoffer.dall@linaro.org>

When an SError arrives during single-step both the SError and debug
exceptions may be pending when the step is completed, and the
architecture doesn't define the ordering of the two.  This means that we
can observe en SError even though we've just completed a step, without
receiving a debug exception.  In that case the DBG_SPSR_SS bit will have
flipped as the instruction executed. After handling the abort in
handle_exit() we test to see if the bit is clear and we were
single-stepping before deciding if we need to exit to user space.
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>
Reviewed-by: NChristoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Signed-off-by: NChristoffer Dall <christoffer.dall@linaro.org>

If we are using guest debug to single-step the guest, we need to ensure
that we exit after emulating the instruction. This only affects
instructions completely emulated by the kernel. For instructions
emulated in userspace, we need to exit and return to complete the
emulation.

The kvm_arm_handle_step_debug() helper sets up the necessary exit
state if needed.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NChristoffer Dall <christoffer.dall@linaro.org>

After emulating instructions we may want return to user-space to handle
single-step debugging. Introduce a helper function, which, if
single-step is enabled, sets the run structure for return and returns
true.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NChristoffer Dall <christoffer.dall@linaro.org>

VTTBR_BADDR_MASK is used to sanity check the size and alignment of the
VTTBR address. It seems to currently be off by one, thereby only
allowing up to 39-bit addresses (instead of 40-bit) and also
insufficiently checking the alignment. This patch fixes it.

This patch is the 32bit pendent of Kristina's arm64 fix, and
she deserves the actual kudos for pinpointing that one.

Fixes: f7ed45be ("KVM: ARM: World-switch implementation")
Cc: <stable@vger.kernel.org> # 3.9
Reported-by: NKristina Martsenko <kristina.martsenko@arm.com>
Reviewed-by: NChristoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NChristoffer Dall <christoffer.dall@linaro.org>

VTTBR_BADDR_MASK is used to sanity check the size and alignment of the
VTTBR address. It seems to currently be off by one, thereby only
allowing up to 47-bit addresses (instead of 48-bit) and also
insufficiently checking the alignment. This patch fixes it.

As an example, with 4k pages, before this patch we have:

  PHYS_MASK_SHIFT = 48
  VTTBR_X = 37 - 24 = 13
  VTTBR_BADDR_SHIFT = 13 - 1 = 12
  VTTBR_BADDR_MASK = ((1 << 35) - 1) << 12 = 0x00007ffffffff000

Which is wrong, because the mask doesn't allow bit 47 of the VTTBR
address to be set, and only requires the address to be 12-bit (4k)
aligned, while it actually needs to be 13-bit (8k) aligned because we
concatenate two 4k tables.

With this patch, the mask becomes 0x0000ffffffffe000, which is what we
want.

Fixes: 0369f6a3 ("arm64: KVM: EL2 register definitions")
Cc: <stable@vger.kernel.org> # 3.11.x
Reviewed-by: NSuzuki K Poulose <suzuki.poulose@arm.com>
Reviewed-by: NChristoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: NKristina Martsenko <kristina.martsenko@arm.com>
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NChristoffer Dall <christoffer.dall@linaro.org>

If set_thread_tidr() is called twice for same task_struct then it will
allocate a new tidr value to it leaving the previous value still
dangling in the vas_thread_ida table.

To fix this the patch changes set_thread_tidr() to check if a tidr
value is already assigned to the task_struct and if yes then returns
zero.

Fixes: ec233ede("powerpc: Add support for setting SPRN_TIDR")
Signed-off-by: NVaibhav Jain <vaibhav@linux.vnet.ibm.com>
Reviewed-by: NAndrew Donnellan <andrew.donnellan@au1.ibm.com>
[mpe: Modify to return 0 in the success case, not the TID value]
Signed-off-by: NMichael Ellerman <mpe@ellerman.id.au>

There is an unsafe signed to unsigned conversion in set_thread_tidr()
that may cause an error value to be assigned to SPRN_TIDR register and
used as thread-id.

The issue happens as assign_thread_tidr() returns an int and
thread.tidr is an unsigned-long. So a negative error code returned
from assign_thread_tidr() will fail the error check and gets assigned
as tidr as a large positive value.

To fix this the patch assigns the return value of assign_thread_tidr()
to a temporary int and assigns it to thread.tidr iff its '> 0'.

The patch shouldn't impact the calling convention of set_thread_tidr()
i.e all -ve return-values are error codes and a return value of '0'
indicates success.

Fixes: ec233ede("powerpc: Add support for setting SPRN_TIDR")
Signed-off-by: NVaibhav Jain <vaibhav@linux.vnet.ibm.com>
Reviewed-by: Christophe Lombard clombard@linux.vnet.ibm.com
Signed-off-by: NMichael Ellerman <mpe@ellerman.id.au>

This was removed from the other architectures in commit
952111d7 ("arch: Remove spin_unlock_wait() arch-specific
definitions").  That landed between when we got upstream and when our
patches were reviewed, so this is a followup patch.
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

This is just a comment change, but it's one that bit me on the mailing
list.  It turns out that issuing a `sfence.vma` enforces instruction
cache ordering in addition to TLB ordering.  This isn't explicitly
called out in the ISA manual, but Andrew will be making that more clear
in a future revision.

CC: Andrew Waterman <andrew@sifive.com>
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

This was just incorrect in the original version.
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

I mis-read the documentation.  After looking at it again the
documentation is actually as clear as it can be, it's just that I didn't
actually read it in order and therefor did the wrong thing.
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

These are obselete.
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

These duplicate the asm-generic definitions are therefor aren't useful.
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

This is another memory model FIXME.
Signed-off-by: NPalmer Dabbelt <palmer@sifive.com>

Our atomics are generated from a complicated series of preprocessor
macros, each of which is slightly different from the last.  When writing
the macros I'd accidentally left some unused arguments floating around.
This patch removes the unused macro arguments.
Signed-off-by: NPalmer Dabbelt <palmer@dabbelt.com>