Meelis Roos reported that kernels built with gcc-4.9 do not boot, we
eventually narrowed this down to only impacting machines using
UltraSPARC-III and derivitive cpus.

The crash happens right when the first user process is spawned:

[   54.451346] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004
[   54.451346]
[   54.571516] CPU: 1 PID: 1 Comm: init Not tainted 3.16.0-rc2-00211-gd7933ab7 #96
[   54.666431] Call Trace:
[   54.698453]  [0000000000762f8c] panic+0xb0/0x224
[   54.759071]  [000000000045cf68] do_exit+0x948/0x960
[   54.823123]  [000000000042cbc0] fault_in_user_windows+0xe0/0x100
[   54.902036]  [0000000000404ad0] __handle_user_windows+0x0/0x10
[   54.978662] Press Stop-A (L1-A) to return to the boot prom
[   55.050713] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004

Further investigation showed that compiling only per_cpu_patch() with
an older compiler fixes the boot.

Detailed analysis showed that the function is not being miscompiled by
gcc-4.9, but it is using a different register allocation ordering.

With the gcc-4.9 compiled function, something during the code patching
causes some of the %i* input registers to get corrupted.  Perhaps
we have a TLB miss path into the firmware that is deep enough to
cause a register window spill and subsequent restore when we get
back from the TLB miss trap.

Let's plug this up by doing two things:

1) Stop using the firmware stack for client interface calls into
   the firmware.  Just use the kernel's stack.

2) As soon as we can, call into a new function "start_early_boot()"
   to put a one-register-window buffer between the firmware's
   deepest stack frame and the top-most initial kernel one.
Reported-by: NMeelis Roos <mroos@linux.ee>
Tested-by: NMeelis Roos <mroos@linux.ee>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The following patch adds support for correctly
recognising M6 and M7 cpu type.
Signed-off-by: NAllen Pais <allen.pais@oracle.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The following patch adds support for correctly
recognizing SPARC-X chips.

cpu : Unknown SUN4V CPU
fpu : Unknown SUN4V FPU
pmu : Unknown SUN4V PMU
Signed-off-by: NKatayama Yoshihiro <kata1@jp.fujitsu.com>
Signed-off-by: NAllen Pais <allen.pais@oracle.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This adds optimized memset/bzero/page-clear routines for Niagara-4.

We basically can do what powerpc has been able to do for a decade (via
the "dcbz" instruction), which is use cache line clearing stores for
bzero and memsets with a 'c' argument of zero.

As long as we make the cache initializing store to each 32-byte
subblock of the L2 cache line, it works.

As with other Niagara-4 optimized routines, the key is to make sure to
avoid any usage of the %asi register, as reads and writes to it cost
at least 50 cycles.

For the user clear cases, we don't use these new routines, we use the
Niagara-1 variants instead.  Those have to use %asi in an unavoidable
way.

A Niagara-4 8K page clear costs just under 600 cycles.

Add definitions of the MRU variants of the cache initializing store
ASIs.  By default, cache initializing stores install the line as Least
Recently Used.  If we know we're going to use the data immediately
(which is true for page copies and clears) we can use the Most
Recently Used variant, to decrease the likelyhood of the lines being
evicted before they get used.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

		Before		After
		--------------	--------------
bw_tcp:         1288.53 MB/sec	1637.77 MB/sec
bw_pipe:        1517.18 MB/sec	2107.61 MB/sec
bw_unix:        1838.38 MB/sec	2640.91 MB/sec

make -s -j128
allmodconfig	5min 49sec	5min 31sec
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

To allow us to add ttable_32.S
Signed-off-by: NSam Ravnborg <sam@ravnborg.org>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Recognize T4 and T5 chips.  Treating them both as "T2 plus other
stuff" should be extremely safe and make sure distributions will work
when those chips actually ship to customers.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Don't use floating point on Niagara2, use the traditional
plain Niagara code instead.

Unroll Niagara loops to 128 bytes for copy, and 256 bytes
for clear.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The cpu compatible string we look for is "SPARC-T3".

As far as memset/memcpy optimizations go, we treat this chip the same
as Niagara-T2/T2+.  Use cache initializing stores for memset, and use
perfetch, FPU block loads, cache initializing stores, and block stores
for copies.

We use the Niagara-T2 perf support, since T3 is a close relative in
this regard.  Later we'll add support for the new events T3 can
report, plus enable T3's new "sample" mode.

For now I haven't added any new ELF hwcap flags.  We probably need
to add a couple, for example:

T2 and T3 both support the population count instruction in hardware.

T3 supports VIS3 instructions, including support (finally) for
partitioned shift.  One can also now move directly between float
and integer registers.

T3 supports instructions meant to help with Galois Field and other HPC
calculations, such as XOR multiply.  Also there are "OP and negate"
instructions, for example "fnmul" which is multiply-and-negate.

T3 recognizes the transactional memory opcodes, however since
transactional memory isn't supported: 1) 'commit' behaves as a NOP and
2) 'chkpt' always branches 3) 'rdcps' returns all zeros and 4) 'wrcps'
behaves as a NOP.

So we'll need about 3 new elf capability flags in the end to represent
all of these things.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Fixes generated by 'codespell' and manually reviewed.
Signed-off-by: NLucas De Marchi <lucas.demarchi@profusion.mobi>

Surprisingly this actually makes LOAD_PER_CPU_BASE() a little
more efficient.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The section .text.init.refok is deprecated and __REF (.ref.text)
should be used in assembly files instead.  This patch cleans up a few
uses of .text.init.refok in the sparc architecture.

Also fix a reference to .text.init in a comment that wasn't updated to
.init.text.
Signed-off-by: NTim Abbott <tabbott@mit.edu>
Cc: David S. Miller <davem@davemloft.net>
Acked-by: NSam Ravnborg <sam@ravnborg.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Fix misspelling of firmware.
Signed-off-by: NNick Andrew <nick@nick-andrew.net>
Signed-off-by: NJiri Kosina <jkosina@suse.cz>

This is an implementation of a suggestion made by Chris Torek:
--------------------
Something else I noticed in passing: the EX and EX_LD/EX_ST macros
scattered throughout the various .S files make a fair bit of .fixup
code, all of which does the same thing.  At the cost of one symbol
in copy_in_user.S, you could just have one common two-instruction
retl-and-mov-1 fixup that they all share.
--------------------

The following is with a defconfig build:

   text	   data	    bss	    dec	    hex	filename
3972767	 344024	 584449	4901240	 4ac978	vmlinux.orig
39688877	 344024	 584449	4897360	 4aba50	vmlinux
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

o Move all files from sparc64/kernel/ to sparc/kernel
  - rename as appropriate
o Update sparc/Makefile to the changes
o Update sparc/kernel/Makefile to include the sparc64 files

NOTE: This commit changes link order on sparc64!

Link order had to change for either of sparc32 and sparc64.
And assuming sparc64 see more testing than sparc32 change link
order on sparc64 where issues will be caught faster.
Signed-off-by: NSam Ravnborg <sam@ravnborg.org>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

So that we can profile code even in a local_irq_disable() section,
only write 14 (instead of 15) into the %pil register to disable IRQs.

This allows PIL level 15 to serve as a pseudo NMI.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Signed-off-by: NDavid S. Miller <davem@davemloft.net>

entry.S was a hodge-podge of several totally unrelated
sets of assembler routines, ranging from FPU trap handlers
to hypervisor call functions.

Split it up into topic-sized pieces.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Currently kernel images are limited to 8MB in size, and this causes
problems especially when enabling features that take up a lot of
kernel image space such as lockdep.

The code now will align the kernel image size up to 4MB and map that
many locked TLB entries.  So, the only practical limitation is the
number of available locked TLB entries which is 16 on Cheetah and 64
on pre-Cheetah sparc64 cpus.  Niagara cpus don't actually have hw
locked TLB entry support.  Rather, the hypervisor transparently
provides support for "locked" TLB entries since it runs with physical
addressing and does the initial TLB miss processing.

Fully utilizing this change requires some help from SILO, a patch for
which will be submitted to the maintainer.  Essentially, SILO will
only currently map up to 8MB for the kernel image and that needs to be
increased.

Note that neither this patch nor the SILO bits will help with network
booting.  The openfirmware code will only map up to a certain amount
of kernel image during a network boot and there isn't much we can to
about that other than to implemented a layered network booting
facility.  Solaris has this, and calls it "wanboot" and we may
implement something similar at some point.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The early per-cpu handling needs a slight tweak to work when booting
on a non-zero cpu.

We got away with this for a long time, but can't any longer as now
even printk() calls functions (cpu_clock() for example) that thus make
early references to per-cpu variables.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

As noted by Al Viro, when we try to call prom_set_trap_table()
in the SMP trampoline code we try to take the PROM call spinlock
which doesn't work because the current thread pointer isn't
valid yet and lockdep depends upon that being correct.

Furthermore, we cannot set the current thread pointer register
because it can't be properly dereferenced until we return from
prom_set_trap_table().  Kernel TLB misses only work after that
call.

So do the PROM call to set the trap table directly instead of
going through the OBP library C code, and thus avoid the lock
altogether.

These calls are guarenteed to be serialized fully.

Since there are now no calls to the prom_set_trap_table{_sun4v}()
library functions, they can be deleted.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This register is not a part of the sun4v architecture.

Niagara 1 and 2 happened to leave it around.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The bzero/memset implementation stays the same as Niagara-1.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Check the cpu type in the OBP device tree before committing to
using the optimized Niagara memcpy and memset implementation.

If we don't recognize the cpu type, use a completely generic
version.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

We can't mark the whole thing init because there are dependencies
in bootloaders that assume that _start, or whatever the image
entry value, is 2 instructions before the "HdrS" signature.

In fact, TILO assumes this entry is always at 0x4000, yikes!

Also, right after the bootloader info area there are OBP strings and
values that get used later in the boot process, and those are not all
provably .init yet.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

We try to fetch the CIF entry pointer from %o4, but that
can get clobbered by the early OBP calls.  It is saved
in %l7 already, so actually this "mov %o4, %l7" can just
be completely removed with no other changes.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

1) The TSB lookup was not using the correct hash mask.

2) It was not aligned on a boundary equal to it's size,
   which is required by the sun4v Hypervisor.

wasn't having it's return value checked, and that bug will be fixed up
as well in a subsequent changeset.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Cheetah systems can have cpuids as large as 1023, although physical
systems don't have that many cpus.

Only three limitations existed in the kernel preventing arbitrary
NR_CPUS values:

1) dcache dirty cpu state stored in page->flags on
   D-cache aliasing platforms.  With some build time
   calculations and some build-time BUG checks on
   page->flags layout, this one was easily solved.

2) The cheetah XCALL delivery code could only handle
   a cpumask with up to 32 cpus set.  Some simple looping
   logic clears that up too.

3) thread_info->cpu was a u8, easily changed to a u16.

There are a few spots in the kernel that still put NR_CPUS
sized arrays on the kernel stack, but that's not a sparc64
specific problem.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

It branches around some necessary prom calls, which we would
need to do even if we are mapped at the correct location already.
So it doesn't work.

The idea was that this sort of thing could be used for the eventual
kexec implementation, but it is clear that this will need to be
done differently.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Else we trigger the new irqs_disable() assertion in start_kernel().
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Signed-off-by: NJörn Engel <joern@wohnheim.fh-wedel.de>
Signed-off-by: NAdrian Bunk <bunk@stusta.de>

Uses of smp_processor_id() get pushed earlier and earlier in
the start_kernel() sequence.  So just get it working before
we call start_kernel() to avoid all possible problems.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Otherwise with too much stuff enabled in the kernel config
we can end up with an unaligned trap table.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Signed-off-by: NDavid S. Miller <davem@davemloft.net>

prom_sun4v_name should be "sun4v" not "SUNW,sun4v"

Also, this is too early to make use of the
.sun4v_Xinsn_patch code patching, so just check
things manually.

This gets us at least to prom_init() on Niagara.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This is where the virtual address of the fault status
area belongs.

To set it up we don't make a hypervisor call, instead
we call OBP's SUNW,set-trap-table with the real address
of the fault status area as the second argument.  And
right before that call we write the virtual address into
ASI_SCRATCHPAD vaddr 0x0.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

We look for "SUNW,sun4v" in the 'compatible' property
of the root OBP device tree node.

Protect every %ver register access, to make sure it is
not touched on sun4v, as %ver is hyperprivileged there.

Lock kernel TLB entries using hypervisor calls instead of
calls into OBP.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Sun4v has 4 interrupt queues: cpu, device, resumable errors,
and non-resumable errors.  A set of head/tail offset pointers
help maintain a work queue in physical memory.  The entries
are 64-bytes in size.

Each queue is allocated then registered with the hypervisor
as we bring cpus up.

The two error queues each get a kernel side buffer that we
use to quickly empty the main interrupt queue before we
call up to C code to log the event and possibly take evasive
action.
Signed-off-by: NDavid S. Miller <davem@davemloft.net>