This follows the ARM change c0177800
("ARM: 6379/1: Assume new page cache pages have dirty D-cache") for the
same rationale:

    There are places in Linux where writes to newly allocated page
    cache pages happen without a subsequent call to flush_dcache_page()
    (several PIO drivers including USB HCD). This patch changes the
    meaning of PG_arch_1 to be PG_dcache_clean and always flush the
    D-cache for a newly mapped page in update_mmu_cache().

This addresses issues seen with executing binaries from MMC, in
addition to some of the other HCDs that don't explicitly do cache
management for their pipe-in buffers.
Requested-by: NYoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

The old ctrl in/out routines are non-portable and unsuitable for
cross-platform use. While drivers/sh has already been sanitized, there
is still quite a lot of code that is not. This converts the arch/sh/ bits
over, which permits us to flag the routines as deprecated whilst still
building with -Werror for the architecture code, and to ensure that
future users are not added.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Now that cached_to_uncached works as advertized in 32-bit mode and we're
never going to be able to map < 16MB anyways, there's no need for the
special uncached section. Kill it off.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

If the page is not mapped into any process's address space then aliases
cannot exist in the cache. So reduce the amount of flushing we perform.
Signed-off-by: NMatt Fleming <matt@console-pimps.org>

It does not make sense to compare virtual and physical addresses for
aliasing, only virtual addresses can be compared for aliases.
Signed-off-by: NMatt Fleming <matt@console-pimps.org>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

When flushing/invalidating the icache/dcache via the memory-mapped IC/OC
address arrays, the associative bit should only be used in conjunction with
virtual addresses. However, we currently flush cache lines based on physical
address, so stop using the associative bit.

It is a better strategy to use non-associative writes (and physical tags) for
flushing the caches anyway, because flushing by virtual address (as with the
A-bit set) requires a valid TLB entry for that virtual address. If one does not
exist in the TLB no exception is generated and the flush is silently ignored.

This is also future-proofing for SH-4A parts which are gradually phasing out
associative writes to the cache array due to the aforementioned case of certain
flushes silently turning in to nops.
Signed-off-by: NMatt Fleming <matt@console-pimps.org>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

The icache may also contain aliases so we must account for them just
like we do when manipulating the dcache. We usually get away with
aliases in the icache because the instructions that are read from memory
are read-only, i.e. they never change. However, the place where this
bites us is when the code has been modified.
Signed-off-by: NMatt Fleming <matt@console-pimps.org>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

The variable 'phys' already contains the physical address to flush. It
is not a virtual address and should not be passed to virt_to_phys().
Signed-off-by: NMatt Fleming <matt@console-pimps.org>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Presently The SH-4 cache flushing code uses flush_cache_4096() for most
of the real flushing work, which breaks down to a fixed 4096 unroll and
increment. Not only is this sub-optimal for larger page sizes, it's also
uncovered a bug in sh4_flush_dcache_page() when large page sizes are used
and we have no cache aliases -- resulting in only a part of the page's
D-cache lines being written back.
Signed-off-by: NValentin Sitdikov <valentin.sitdikov@siemens.com>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

To allow the MMU to be switched between 29bit and 32bit mode at runtime
some constants need to swapped for functions that return a runtime
value.
Signed-off-by: NMatt Fleming <matt@console-pimps.org>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Replace the use of PHYSADDR() with __pa(). PHYSADDR() is based on the
idea that all addresses in P1SEG are untranslated, so we can access an
address's physical page as an offset from P1SEG. This doesn't work for
CONFIG_PMB/CONFIG_PMB_FIXED because pages in P1SEG and P2SEG are used
for PMB mappings and so can be translated to any physical address.

Likewise, replace a P1SEGADDR() use with virt_to_phys().
Signed-off-by: NMatt Fleming <matt@console-pimps.org>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Fix some callers of jump_to_uncached() and back_to_cached() that were
not annotated with __uses_jump_to_uncached.
Signed-off-by: NMatt Fleming <matt@console-pimps.org>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

If PAGE_SIZE is presently over 4k we do a lot of extra flushing given
that we purge the cache 4k at a time. Make it explicitly 4k per
iteration, rather than iterating for PAGE_SIZE before looping over again.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This builds on top of the MIPS r4k code that does roughly the same thing.
This permits the use of kmap_coherent() for mapped pages with dirty
dcache lines and falls back on kmap_atomic() otherwise.

This also fixes up a problem with the alias check and defers to
shm_align_mask directly.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This kills off the unrolled segment based flushers on SH-4 and switches
over to a generic unrolled approach derived from the writethrough segment
flusher.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

PHYSADDR() runs in to issues in 32-bit mode when we do not have the
legacy P1/P2 areas mapped, as such, we need to use page_to_phys()
directly, which also happens to do the right thing in legacy 29-bit mode.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

The i-cache flush in the case of VM_EXEC was added way back when as a
sanity measure, and in practice we only care about evicting aliases from
the d-cache. As a result, it's possible to drop the i-cache flush
completely here.

After careful profiling it's also come up that all of the work associated
with hunting down aliases and doing ranged flushing ends up generating
more overhead than simply blasting away the entire dcache, particularly
if there are many mm's that need to be iterated over. As a result of
that, just move back to flush_dcache_all() in these cases, which restores
the old behaviour, and vastly simplifies the path.

Additionally, on platforms without aliases at all, this can simply be
nopped out. Presently we have the alias check in the SH-4 specific
version, but this is true for all of the platforms, so move the check up
to a generic location. This cuts down quite a bit on superfluous cacheop
IPIs.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

There was quite a lot of tab->space damage done here from a former patch,
clean it up once and for all.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This reverts commit 64a6d722.

Unfortunately we can't use on_each_cpu() for all of the cache ops, as
some of them only require preempt disabling. This seems to be the same
issue that impacts the mips r4k caches, where this code was based on.
This fixes up a deadlock that showed up in some IRQ context cases.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This adopts the special-cased 2-way write-through dcache flusher for
N-ways and moves it in to the generic path. Assignment is done at runtime
via the check for the CCR_CACHE_WT bit in the same path as the per-way
writeback flushers.
Signed-off-by: NMatt Fleming <matt@console-pimps.org>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Change the method used to flush the cache in write-through mode to
avoid corrupted data being written back to memory.
Signed-off-by: NStuart Menefy <stuart.menefy@st.com>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This is a pure documentation, to try to explain why the cache flushing code
for the SH4 is implemented the way it is.
Signed-off-by: NStuart Menefy <stuart.menefy@st.com>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

on_each_cpu() takes care of IRQ and preempt handling, the localized
handling in each of the called functions can be killed off.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This does a bit of rework for making the cache flushers SMP-aware. The
function pointer-based flushers are renamed to local variants with the
exported interface being commonly implemented and wrapping as necessary.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

mapping is unused on the SMP build, trigger a build error. Move it under
the ifdef.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This paves the way for allowing individual CPUs to overload the
individual flushing routines that they care about without having to
depend on weak aliases. SH-4 is converted over initially, as it wires
up pretty much everything. The majority of the other CPUs will simply use
the default no-op implementation with their own region flushers wired up.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

We use flush_cache_page() outright in copy_to_user_page(), and nothing
else needs it, so just kill it off. SH-5 still defines its own version,
but that too will go away in the same fashion once it converts over.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

flush_dcache_all() is used internally by the SH-4 cache code, it is not
part of the exported cache API, so make it static and don't export it.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This migrates the alias computation and printing of probed cache
parameters from the SH-4 code to the shared cpu_cache_init().

This permits other platforms with aliases to make use of the same
probe logic without having to roll their own, and also produces
consistent output regardless of platform.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This provides a central point for CPU cache initialization routines.
This replaces the antiquated p3_cache_init() method, which the vast
majority of CPUs never cared about.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This optimizes for the cases when a CPU does not yet have a valid ASID
context associated with it, as in this case there is no work for any of
flush_cache_mm()/flush_cache_page()/flush_cache_range() to do. Based on
the the MIPS implementation.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This splits out the SH-4 __flush_xxx_region() functions and defines them
as weak symbols. This allows us to provide optimized versions without
having to ifdef cache-sh4.c to death.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This inverts the delayed dcache flush a bit to be more in line with other
platforms. At the same time this also gives us the ability to do some
more optimizations and cleanup. Now that the update_mmu_cache() callsite
only tests for the bit, the implementation can gradually be split out and
made generic, rather than relying on special implementations for each of
the peculiar CPU types.

SH7705 in 32kB mode and SH-4 still need slightly different handling, but
this is something that can remain isolated in the varying page copy/clear
routines. On top of that, SH-X3 is dcache coherent, so there is no need
to bother with any of these tests in the PTEAEX version of
update_mmu_cache(), so we kill that off too.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

This uses jump_to_uncached() which is now given the noinline attribute
due to the special section mapping. Kill off the inline attribute to
fix up compilation failure.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Add implementation of flush_icache_range() suitable for signal handler
and kprobes. Remove flush_cache_sigtramp() and change signal.c to use
flush_icache_range().
Signed-off-by: NChris Smith <chris.smith@st.com>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Presently most of the 29-bit physical parts do P1/P2 segmentation
with a 1:1 cached/uncached mapping, jumping between the two to
control the caching behaviour. This provides the basic infrastructure
to maintain this behaviour on 32-bit physical parts that don't map
P1/P2 at all, using a shiny new linker section and corresponding
fixmap entry.
Signed-off-by: NStuart Menefy <stuart.menefy@st.com>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Calculate the number of cache aliases on probed L2 caches, and while
we're at it, print out the detected statistics at boot time for these
also.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

There was an off-by-1 on the cache alias detection logic on SH-4,
which caused n_aliases to always be 1 even when the page size
precluded the existence of aliases.

With this corrected, 64KB pages happily reports n_aliases == 0, and
hits the appropriate fast paths in the flushing routines.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

current_cpu_data uses smp_processor_id() in order to find the
corresponding cpu_data. As the cache descs are all currently
identical, just have this look at probed results from the boot
CPU.
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>