These still require more testing, so revert them for now. We keep the
off-by-1 in the fixmap colouring and drop the rest.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

The previous implementation of clear_user_highpage and copy_user_highpage
checked to see if there was a D-cache aliasing issue between the user
and kernel mappings of a page, but if there was they always did a
flush with writeback on the dirtied kernel alias.

However as we now have the ability to map a page into kernel space
with the same cache colour as the user mapping, there is no need to
write back this data.

Currently we also invalidate the kernel alias as a precaution, however
I'm not sure if this is actually required.

Also correct the definition of FIX_CMAP_END so that the mappings created
by kmap_coherent() are actually at the correct colour.
Signed-off-by: NStuart Menefy <stuart.menefy@st.com>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This gets the build fixed up for the sh64 cache enabled case.
Disabling still needs further abstraction for independent I/D-cache
disabling.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

These were previously hidden in sh_ksyms_32, despite also being needed
for sh64 now that the cache.c code is shared.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Add code to handle the cache disabled case. Fixes breakage introduced by
37443ef3 ("sh: Migrate SH-4 cacheflush
ops to function pointers."). Without this patch configuring caches off
with CONFIG_CACHE_OFF=y makes kfr2r09 and migo-r lock up in fbdev
deferred io or early user space.
Signed-off-by: NMagnus Damm <damm@opensource.se>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This too follows the ARM change, given that the issue at hand applies to
all platforms that implement lazy D-cache writeback.

This fixes up the case when a page mapping disappears between the
flush_dcache_page() call (when PG_dcache_dirty is set for the page) and
the update_mmu_cache() call -- such as in the case of swap cache being
freed early. This kills off the mapping test in update_mmu_cache() and
switches to simply testing for PG_dcache_dirty.
Reported-by: NNitin Gupta <ngupta@vflare.org>
Reported-by: NHugh Dickins <hugh.dickins@tiscali.co.uk>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

The i-cache flush in the case of VM_EXEC was added way back when as a
sanity measure, and in practice we only care about evicting aliases from
the d-cache. As a result, it's possible to drop the i-cache flush
completely here.

After careful profiling it's also come up that all of the work associated
with hunting down aliases and doing ranged flushing ends up generating
more overhead than simply blasting away the entire dcache, particularly
if there are many mm's that need to be iterated over. As a result of
that, just move back to flush_dcache_all() in these cases, which restores
the old behaviour, and vastly simplifies the path.

Additionally, on platforms without aliases at all, this can simply be
nopped out. Presently we have the alias check in the SH-4 specific
version, but this is true for all of the platforms, so move the check up
to a generic location. This cuts down quite a bit on superfluous cacheop
IPIs.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This fixes up a number of outstanding issues observed with old mappings
on the same colour hanging around. This requires some more optimal
handling, but is a safe fallback until all of the corner cases have been
handled.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This fixes up the kmap_coherent/kunmap_coherent() interface for recent
changes both in the page fault path and the shared cache flushers, as
well as adding in some optimizations.

One of the key things to note here is that the TLB flush itself is
deferred until the unmap, and the call in to update_mmu_cache() itself
goes away, relying on the regular page fault path to handle the lazy
dcache writeback if necessary.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This builds on top of the previous reversion and implements a special
on_each_cpu() variant that simple disables preemption across the call
while leaving the interrupt state to the function itself. There were some
unintended consequences with IRQ disabling in some of these paths on UP
that ran in to a deadlock scenario with IRQs being missed.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This does a bit of rework for making the cache flushers SMP-aware. The
function pointer-based flushers are renamed to local variants with the
exported interface being commonly implemented and wrapping as necessary.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Now that the SH-5 code is more or less behaving with the new cacheflush
interface, wire up the initialization code.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This paves the way for allowing individual CPUs to overload the
individual flushing routines that they care about without having to
depend on weak aliases. SH-4 is converted over initially, as it wires
up pretty much everything. The majority of the other CPUs will simply use
the default no-op implementation with their own region flushers wired up.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This migrates the alias computation and printing of probed cache
parameters from the SH-4 code to the shared cpu_cache_init().

This permits other platforms with aliases to make use of the same
probe logic without having to roll their own, and also produces
consistent output regardless of platform.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This provides a central point for CPU cache initialization routines.
This replaces the antiquated p3_cache_init() method, which the vast
majority of CPUs never cared about.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This builds in the newly created cache.c (renamed from pg-mmu.c) for both
MMU and NOMMU configurations. The kmap_coherent() stubs and alias
information recorded by each CPU family takes care of doing the right
thing while enabling the code to be commonly shared.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This plugs in kmap_coherent() for the non-SH4 cases to permit the
pg-mmu.c bits to be used generically across all CPUs. SH-5 is still in
the TODO state, but will move over to fixmap and the generic interface
gradually.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This kills off the ifdef from kmap_coherent_init() and just bails if
there are no cache aliases. This permits the kmap coherent code to be
used on other CPUs.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This provides a __flush_anon_page() that handles both the aliasing and
non-aliasing cases. This fixes up some crashes with heavy
get_user_pages() users.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

kunmap_coherent() doesn't do anything with its arguments, so just kill
them off.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

kmap_coherent() has gotten too big to leave as an inline, so we
bring it out-of-line.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This plugs in a BUG_ON() in kmap_coherent() for PG_dcache_dirty pages
to catch when things go horribly wrong. Copied from the MIPS
implementation.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This splits out a separate __update_cache()/__update_tlb() for
update_mmu_cache() to wrap in to. This lets us share the common
__update_cache() bits while keeping special __update_tlb() handling
broken out.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Now that the SH-4 page clear/copy ops are generic, they can be used for
all platforms with CONFIG_MMU=y. SH-5 remains the odd one out, but it too
will gradually be converted over to using this interface.

SH-3 platforms which do not contain aliases will see no impact from this
change, while aliasing SH-3 platforms will get the same interface as
SH-4.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This wires up clear_user_highpage() on SH-4 and subsequently converts the
SH7705 32kB cache mode over to using it. Now that the SH-4 implementation
handles all of the dcache purging directly in the aliasing case, there is
no need to do this in the default clear_page() implementation.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This inverts the delayed dcache flush a bit to be more in line with other
platforms. At the same time this also gives us the ability to do some
more optimizations and cleanup. Now that the update_mmu_cache() callsite
only tests for the bit, the implementation can gradually be split out and
made generic, rather than relying on special implementations for each of
the peculiar CPU types.

SH7705 in 32kB mode and SH-4 still need slightly different handling, but
this is something that can remain isolated in the varying page copy/clear
routines. On top of that, SH-X3 is dcache coherent, so there is no need
to bother with any of these tests in the PTEAEX version of
update_mmu_cache(), so we kill that off too.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

There was a race in the kmap_coherent() implementation. While we
guarded against preemption, there was nothing preventing eviction of
the pre-faulted fixmap entry from the UTLB. Under certain workloads
this would result in the fixmap entries used for cache colouring being
evicted from the UTLB in the midst of a copy_page().

In addition to pre-faulting, we also make sure to preserve the PTEs
in the kernel page table and introduce a cached PTE for kmap_coherent()
usage. This follows a similar change on MIPS ("[MIPS] Fix aliasing bug
in copy_to_user_page / copy_from_user_page").
Reported-by: NHideo Saito <saito@densan.co.jp>
Reported-by: NCHIKAMA Masaki <masaki.chikama@gmail.com>
Tested-by: NYoshihiro Shimoda <shimoda.yoshihiro@renesas.com>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Remove inline from ptep_get_and_clean() to match with header file prototype.
Makes linux-next build.
Signed-off-by: NMagnus Damm <damm@igel.co.jp>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This moves copy_{to,from}_user_page() out-of-line on SH-4 and
converts for the kmap_coherent() API. Based on the MIPS
implementation.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

With the kmap_coherent() API in place, this is trivial to implement,
and lets us avoid the cache flush in certain cases.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This wires up kmap_coherent() and kunmap_coherent() on SH-4, and
moves away from the p3map_mutex and reserved P3 space, opting to
use fixmaps for colouring instead.

The copy_user_page()/clear_user_page() implementations are moved
to this, which fixes the nasty blowups with spinlock debugging
as a result of having some of these calls nested under the page
table lock.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

These ended up causing too many problems on older parts,
revert for now..
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Rename the existing flush routines to local_ variants for use by
the IPI-backed global flush routines on SMP.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

There are a lot of bogus cpu_data-> references that only end up working
for the boot CPU, convert these to current_cpu_data to fixup SMP.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This converts the lazy dcache handling to the model described in
Documentation/cachetlb.txt and drops the ptep_get_and_clear() hacks
used for the aliasing dcaches on SH-4 and SH7705 in 32kB mode. As a
bonus, this slightly cuts down on the cache flushing frequency.

With that and the PTEA handling out of the way, the update_mmu_cache()
implementations can be consolidated, and we no longer have to worry
about which configuration the cache is in for the SH7705 case.

And finally, explicitly disable the lazy writeback on SMP (SH-4A).
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>