With more recent compilers inline doesn't necessarily means a function
will always be inlined.  So leave that decission to the compiler and
make the function as __init.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

c-r4k.c and c-sb1.c use drop_mmu_context() to flush virtually tagged
I-caches, but this does not work for flushing other task's icache.  This
is for example triggered by copy_to_user_page() called from ptrace(2).
Use indexed flush for such cases.
Signed-off-by: NAtsushi Nemoto <anemo@mba.ocn.ne.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

On the 34K the redundant cache operations were causing excessive stalls
resulting in realtime code running on the second VPE missing its deadline.
For all other platforms this patch is just a significant performance
improvment as illustrated by below benchmark numbers.

Processor, Processes - times in microseconds - smaller is better
------------------------------------------------------------------------------
Host                 OS  Mhz null null      open slct sig  sig  fork exec sh
                             call  I/O stat clos TCP  inst hndl proc proc proc
--------- ------------- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
25Kf      2.6.18-rc4     533 0.49 1.16 7.57 33.4 30.5 1.34 12.4 5497 17.K 54.K
25Kf      2.6.18-rc4-p   533 0.49 1.16 6.68 23.0 30.7 1.36 8.55 5030 16.K 48.K
4Kc       2.6.18-rc4      80 4.21 15.0 131. 289. 261. 16.5 258. 18.K 70.K 227K
4Kc       2.6.18-rc4-p    80 4.34 13.1 128. 285. 262. 18.2 258. 12.K 52.K 176K
34Kc      2.6.18-rc4      40 5.01 14.0 61.6 90.0 477. 17.9 94.7 29.K 108K 342K
34Kc      2.6.18-rc4-p    40 4.98 13.9 61.2 89.7 475. 17.6 93.7 8758 44.K 158K
BCM1480   2.6.18-rc4     700 0.28 0.60 3.68 5.92 16.0 0.78 5.08 931. 3163 15.K
BCM1480   2.6.18-rc4-p   700 0.28 0.61 3.65 5.85 16.0 0.79 5.20 395. 1464 8385
TX49-16K  2.6.18-rc3     197 0.73 2.41 19.0 37.8 82.9 2.94 17.5 4438 14.K 56.K
TX49-16K  2.6.18-rc3-p   197 0.73 2.40 19.9 36.3 82.9 2.94 23.4 2577 9103 38.K
TX49-32K  2.6.18-rc3     396 0.36 1.19 6.80 11.8 41.0 1.46 8.17 2738 8465 32.K
TX49-32K  2.6.18-rc3-p   396 0.36 1.19 6.82 10.2 41.0 1.46 8.18 1330 4638 18.K
    
Original patch by me with enhancements by Atsushi Nemoto.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>
Signed-off-by: NAtsushi Nemoto <anemo@mba.ocn.ne.jp>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NYoichi Yuasa <yoichi_yuasa@tripeaks.co.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NYoichi Yuasa <yoichi_yuasa@tripeaks.co.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NYoichi Yuasa <yoichi_yuasa@tripeaks.co.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NJörn Engel <joern@wohnheim.fh-wedel.de>
Signed-off-by: NAdrian Bunk <bunk@stusta.de>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Don't ask.
Signed-off-by: NChris Dearman <chris@mips.com>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NChris Dearman <chris@mips.com>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

    
Save the Config.OD bit from being clobbered by coherency_setup(). This
bit, when set, fixes various errata in the early steppings of Au1x00
SOCs.  Unfortunately, the bit was write-only on the most early of them.
In addition, also restore the bit after a wakeup from sleep.
Signed-off-by: NSergei Shtylyov <sshtylyov@ru.mvista.com>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NJoshua Kinard <kumba@gentoo.org>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

A proper fix would involve introducing the notion of shared caches but
at this stage of 2.6.17 that's going to be too intrusive and not needed
for current hardware; aside I think some discussion will be needed.

So for now on the affected SMP configurations which happen to suffer from
cache aliases we make use of the fact that a single cache will be shared
by all processors.  This solves the deadlock issue and will improve
performance by getting rid of the smp_call_function overhead.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

    
The 34K is very much like a 24K on steroids.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NAtsushi Nemoto <anemo@mba.ocn.ne.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Fix the cache index value in tx49_blast_icache32_page_indexed().
This is a damage by de62893b commit.
Signed-off-by: NAtsushi Nemoto <anemo@mba.ocn.ne.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

    
The TX49XX has the prefetch instruction.  It supports only Pref_Load
(hint 0).  Actually changes in this patch except for Kconfig are not
have any effects, I added these changes to prevent misuse of unsupported
hints.
Signed-off-by: NAtsushi Nemoto <anemo@mba.ocn.ne.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

    
If dcache_size != icache_size or dcache_size != scache_size, or
set-associative cache, icache/scache does not flushed properly.  Make
blast_?cache_page_indexed() masks its index value correctly.  Also,
use physical address for physically indexed pcache/scache.
Signed-off-by: NAtsushi Nemoto <anemo@mba.ocn.ne.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

    
When a CPU has no scache, the scache flushing functions currently
aren't getting initialized and the NULL pointer is eventually called
as a function.  Initialize the scache flushing functions as a noop
when there's no scache.
    
Initial patch by me and most of the debugging done by Martin Michlmayr.
Signed-off-by: NMartin Michlmayr <tbm@cyrius.com>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

    
Add blast_xxx_range(), protected_blast_xxx_range() etc. for common
use.  They are built by __BUILD_BLAST_CACHE_RANGE().
Use protected_cache_op() macro for various protected_ routines.
Output code should be logically same.
Signed-off-by: NAtsushi Nemoto <anemo@mba.ocn.ne.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

    
This fixes sparse warnings 'dereference of noderef expression'.
Signed-off-by: NAtsushi Nemoto <anemo@mba.ocn.ne.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NThiemo Seufer <ths@networkno.de>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NThiemo Seufer <ths@networkno.de>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NThiemo Seufer <ths@networkno.de>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NThiemo Seufer <ths@networkno.de>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

will also avoid smp_call_function from doing stupid things when called
from a CPU that is not yet marked online.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

a little polishing.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NThiemo Seufer <ths@networkno.de>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NThiemo Seufer <ths@networkno.de>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>