Commit 8263db4d ("MIPS: cmpxchg: Implement __cmpxchg() as a
function") refactored our implementation of __cmpxchg() to be a function
rather than a macro, with the aim of making it easier to read & modify.
Unfortunately the commit breaks use of cmpxchg() for signed 32 bit
values when we have a 64 bit kernel with kernel_uses_llsc == false,
because:

 - In cmpxchg_local() we cast the old value to the type the pointer
   points to, and then to an unsigned long. If the pointer points to a
   signed type smaller than 64 bits then the old value will be sign
   extended to 64 bits. That is, bits beyond the size of the pointed to
   type will be set to 1 if the old value is negative. In the case of a
   signed 32 bit integer with a negative value, bits 63:32 will all be
   set.

 - In __cmpxchg_asm() we load the value from memory, ie. dereference the
   pointer, and store the value as an unsigned integer (__ret) whose
   size matches the pointer. For a 32 bit cmpxchg() this means we store
   the value in a u32, because the pointer provided to __cmpxchg_asm()
   by __cmpxchg() is of type volatile u32 *.

 - __cmpxchg_asm() then checks whether the value in memory (__ret)
   matches the provided old value, by comparing the two values. This
   results in the u32 being promoted to a 64 bit unsigned long to match
   the old argument - however because both types are unsigned the value
   is zero extended, which does not match the sign extension performed
   on the old value in cmpxchg_local() earlier.

This mismatch means that unfortunate cmpxchg() calls can incorrectly
fail for 64 bit kernels with kernel_uses_llsc == false. This is the case
on at least non-SMP Cavium Octeon kernels, which hardcode
kernel_uses_llsc in their cpu-feature-overrides.h header. Using a
v4.13-rc7 kernel configured using cavium_octeon_defconfig with SMP
manually disabled, this presents itself as oddity when we reach
userland - for example:

  can't run '/bin/mount': Text file busy
  can't run '/bin/mkdir': Text file busy
  can't run '/bin/mkdir': Text file busy
  can't run '/bin/mount': Text file busy
  can't run '/bin/hostname': Text file busy
  can't run '/etc/init.d/rcS': Text file busy
  can't run '/sbin/getty': Text file busy
  can't run '/sbin/getty': Text file busy

It appears that some part of the init process, which is in this case
buildroot's busybox init, is running successfully. It never manages to
reach the login prompt though, and complains about /sbin/getty being
busy repeatedly and indefinitely.

Fix this by casting the old value provided to __cmpxchg_asm() to an
appropriately sized unsigned integer, such that we consistently
zero-extend avoiding the mismatch. The __cmpxchg_small() case for 8 & 16
bit values is unaffected because __cmpxchg_small() already masks
provided values appropriately.
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>
Fixes: 8263db4d ("MIPS: cmpxchg: Implement __cmpxchg() as a function")
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/17226/
Cc: linux-mips@linux-mips.org
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

The __xchg() function declares its first 2 arguments in reverse order
compared to the xchg() macro, which is confusing & serves no purpose.
Reorder the arguments such that __xchg() & xchg() match.
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16356/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Implement support for 1 & 2 byte cmpxchg() using read-modify-write atop
a 4 byte cmpxchg(). This allows us to support these atomic operations
despite the MIPS ISA only providing 4 & 8 byte atomic operations.

This is required in order to support queued rwlocks (qrwlock) in a later
patch, since these make use of a 1 byte cmpxchg() in their slow path.
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16355/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Implement 1 & 2 byte xchg() using read-modify-write atop a 4 byte
cmpxchg(). This allows us to support these atomic operations despite the
MIPS ISA only providing for 4 & 8 byte atomic operations.

This is required in order to support queued spinlocks (qspinlock) in a
later patch, since these make use of a 2 byte xchg() in their slow path.
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16354/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Replace the macro definition of __cmpxchg() with an inline function,
which is easier to read & modify. The cmpxchg() & cmpxchg_local() macros
are adjusted to call the new __cmpxchg() function.
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16353/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

The __xchg_u32() & __xchg_u64() functions now add very little value.
This patch therefore removes them, by:

  - Moving memory barriers out of them & into xchg(), which also removes
    the duplication & readies us to support xchg_relaxed() if we wish to.

  - Calling __xchg_asm() directly from __xchg().

  - Performing the check for CONFIG_64BIT being enabled in the size=8
    case of __xchg().
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16352/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

xchg() has up until now simply returned the x parameter in cases where
it is called with a pointer to a value of an unsupported size. This will
often cause the calling code to hit a failure path, presuming that the
value of x differs from the content of the memory pointed at by ptr, but
we can do better by producing a compile-time or link-time error such
that unsupported calls to xchg() are detectable earlier than runtime.

This patch does this in the same was as is already done for cmpxchg(),
using a call to a missing function annotated with __compiletime_error().
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16351/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Our cmpxchg() implementation relies upon generating a call to a function
which doesn't really exist (__cmpxchg_called_with_bad_pointer) to create
a link failure in cases where cmpxchg() is called with a pointer to a
value of an unsupported size.

The __compiletime_error macro can be used to decorate a function such
that a call to it generates a compile-time, rather than a link-time,
error. This patch uses __compiletime_error to cause bad cmpxchg() calls
to error out at compile time rather than link time, allowing errors to
occur more quickly & making it easier to spot where the problem comes
from.
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16350/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Use a macro to generate the 32 & 64 bit variants of the backing code for
xchg(), much as is already done for cmpxchg(). This removes the
duplication that could previously be found in __xchg_u32() &
__xchg_u64().
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16349/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Prior to this patch the xchg & cmpxchg functions have duplicated code
which is for all intents & purposes identical apart from use of a
branch-likely instruction in the R10000_LLSC_WAR case & a regular branch
instruction in the non-R10000_LLSC_WAR case.

This patch removes the duplication, declaring a __scbeqz macro to select
the branch instruction suitable for use when checking the result of an
sc instruction & making use of it to unify the 2 cases.

In __xchg_u{32,64}() this means writing the branch in asm, where it was
previously being done in C as a do...while loop for the
non-R10000_LLSC_WAR case. As this is a single instruction, and adds
consistency with the R10000_LLSC_WAR cases & the cmpxchg() code, this
seems worthwhile.
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16348/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

We removed the only user of this define in the rtmutex code. Get rid
of it.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

This is in preparation of adding HAVE_VIRT_CPU_ACCOUNTING_GEN support in
the next patch.

Without having cmpxchg64 to use the generic implementation, kernel linking
will complain:

kernel/built-in.o: In function `cputime_adjust':
cputime.c:(.text+0x33748): undefined reference to `__cmpxchg_called_with_bad_pointer'
cputime.c:(.text+0x33810): undefined reference to `__cmpxchg_called_with_bad_pointer'
Signed-off-by: NDeng-Cheng Zhu <dengcheng.zhu@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: macro@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9474/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

MIPS R6 changed the opcodes for LL/SC instructions so we need to set
the correct ISA.

Cc: Matthew Fortune <Matthew.Fortune@imgtec.com>
Signed-off-by: NMarkos Chandras <markos.chandras@imgtec.com>

The GCC_OFF12_ASM macro is used for 12-bit immediate constrains
but we will also use it for 9-bit constrains on MIPS R6 so we
rename it to something more appropriate.

Cc: Maciej W. Rozycki <macro@linux-mips.org>
Signed-off-by: NMarkos Chandras <markos.chandras@imgtec.com>

In the microMIPS encoding some memory access instructions have their
immediate offset reduced to 12 bits only.  That does not match the GCC
`R' constraint we use in some places to satisfy the requirement,
resulting in build failures like this:

{standard input}: Assembler messages:
{standard input}:720: Error: macro used $at after ".set noat"
{standard input}:720: Warning: macro instruction expanded into multiple instructions

Fix the problem by defining a macro, `GCC_OFF12_ASM', that expands to
the right constraint depending on whether microMIPS or standard MIPS
code is produced.  Also apply the fix to where `m' is used as in the
worst case this change does nothing, e.g. where the pointer was already
in a register such as a function argument and no further offset was
requested, and in the best case it avoids an extraneous sequence of up
to two instructions to load the high 20 bits of the address in the LL/SC
loop.  This reduces the risk of lock contention that is the higher the
more instructions there are in the critical section between LL and SC.

Strictly speaking we could just bulk-replace `R' with `ZC' as the latter
constraint adjusts automatically depending on the ISA selected.
However it was only introduced with GCC 4.9 and we keep supporing older
compilers for the standard MIPS configuration, hence the slightly more
complicated approach I chose.

The choice of a zero-argument function-like rather than an object-like
macro was made so that it does not look like a function call taking the
C expression used for the constraint as an argument.  This is so as not
to confuse the reader or formatting checkers like `checkpatch.pl' and
follows previous practice.
Signed-off-by: NMaciej W. Rozycki <macro@codesourcery.com>
Signed-off-by: NSteven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/8482/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

With binutils 2.24 the attempt to switch with microMIPS mode to MIPS III
mode through .set mips3 results in *lots* of warnings like

{standard input}: Assembler messages:
{standard input}:397: Warning: the 64-bit MIPS architecture does not support the `smartmips' extension

during a kernel build.  Fixed by using .set arch=r4000 instead.

This breaks support for building the kernel with binutils 2.13 which
was supported for 32 bit kernels only anyway and 2.14 which was a bad
vintage for MIPS anyway.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Having received another series of whitespace patches I decided to do this
once and for all rather than dealing with this kind of patches trickling
in forever.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Fix the following build breakage in v3.4-rc1:

  CC      kernel/irq_work.o
In file included from include/linux/irq_work.h:4:0,
                 from kernel/irq_work.c:10:
include/linux/llist.h: In function 'llist_del_all':
include/linux/llist.h:178:2: error: implicit declaration of function 'BUILD_BUG_ON' [-Werror=implicit-function-declaration]
Signed-off-by: NAaro Koskinen <aaro.koskinen@iki.fi>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/3568/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Disintegrate asm/system.h for MIPS.
Signed-off-by: NDavid Howells <dhowells@redhat.com>
Acked-by: NRalf Baechle <ralf@linux-mips.org>
cc: linux-mips@linux-mips.org

It was a nice optimization - on paper at least.  In practice it results in
branches that may exceed the maximum legal range for a branch.  We can
fight that problem with -ffunction-sections but -ffunction-sections again
is incompatible with -pg used by the function tracer.

By rewriting the loop around all simple LL/SC blocks to C we reduce the
amount of inline assembler and at the same time allow GCC to often fill
the branch delay slots with something sensible or whatever else clever
optimization it may have up in its sleeve.

With this optimization gone we also no longer need -ffunction-sections,
so drop it.

This optimization was originally introduced in 2.6.21, commit
5999eca25c1fd4b9b9aca7833b04d10fe4bc877d (linux-mips.org) rsp.
f65e4fa8 (kernel.org).

Original fix for the issues which caused me to pull this optimization by
Paul Gortmaker <paul.gortmaker@windriver.com>.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Replace some instances of smp_llsc_mb() with a new macro
smp_mb__before_llsc().  It is used before ll/sc sequences that are
documented as needing write barrier semantics.

The default implementation of smp_mb__before_llsc() is just smp_llsc_mb(),
so there are no changes in semantics.

Also simplify definition of smp_mb(), smp_rmb(), and smp_wmb() to be just
barrier() in the non-SMP case.
Signed-off-by: NDavid Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/851/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

On some CPUs, it is more efficient to disable and enable interrupts in the
kernel rather than use ll/sc for atomic operations.  But if we were to set
cpu_has_llsc to false, we would break the userspace futex interface (in
asm/futex.h).

We separate the two concepts, with a new predicate kernel_uses_llsc, that
lets us disable the kernel's use of ll/sc while still allowing the futex
code to use it.

Also there were a couple of cases in bitops.h where we were using ll/sc
unconditionally even if cpu_has_llsc were false.
Signed-off-by: NDavid Daney <ddaney@caviumnetworks.com>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Make sure that at least cmpxchg64_local is available on all architectures to use
for unsigned long long values.
Signed-off-by: NMathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Cc: Ralf Baechle <ralf@linux-mips.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>