Before we plug in highmem support, some of code needs to be ready for it
 - copy_user_highpage() needs to be using the kmap_atomic API
 - mk_pte() can't assume page_address()
 - do_page_fault() can't assume VMALLOC_END is end of kernel vaddr space
Signed-off-by: NAlexey Brodkin <abrodkin@synopsys.com>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NAlexey Brodkin <abrodkin@synopsys.com>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

In case of ARCv2 CPU there're could be following configurations
that affect cache handling for data exchanged with peripherals
via DMA:
 [1] Only L1 cache exists
 [2] Both L1 and L2 exist, but no IO coherency unit
 [3] L1, L2 caches and IO coherency unit exist

Current implementation takes care of [1] and [2].
Moreover support of [2] is implemented with run-time check
for SLC existence which is not super optimal.

This patch introduces support of [3] and rework of DMA ops
usage. Instead of doing run-time check every time a particular
DMA op is executed we'll have 3 different implementations of
DMA ops and select appropriate one during init.

As for IOC support for it we need:
 [a] Implement empty DMA ops because IOC takes care of cache
     coherency with DMAed data
 [b] Route dma_alloc_coherent() via dma_alloc_noncoherent()
     This is required to make IOC work in first place and also
     serves as optimization as LD/ST to coherent buffers can be
     srviced from caches w/o going all the way to memory
Signed-off-by: NAlexey Brodkin <abrodkin@synopsys.com>
[vgupta:
  -Added some comments about IOC gains
  -Marked dma ops as static,
  -Massaged changelog a bit]
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

SLC maintenance ops need to be serialized by software as there is no
inherent buffering / quequing of aux commands. It can silently ignore a
new aux operation if previous one is still ongoing (SLC_CTRL_BUSY)

So gaurd the SLC op using a spin lock

The spin lock doesn't seem to be contended even in heavy workloads such
as iperf. On FPGA @ 75 MHz.

 [1] Before this change:
 ============================================================
  # iperf -c 10.42.0.1
 ------------------------------------------------------------
 Client connecting to 10.42.0.1, TCP port 5001
 TCP window size: 43.8 KByte (default)
 ------------------------------------------------------------
 [  3] local 10.42.0.110 port 38935 connected with 10.42.0.1 port 5001
 [ ID] Interval       Transfer     Bandwidth
 [  3]  0.0-10.0 sec  48.4 MBytes  40.6 Mbits/sec
 ============================================================

 [2] After this change:
 ============================================================
 # iperf -c 10.42.0.1
 ------------------------------------------------------------
 Client connecting to 10.42.0.1, TCP port 5001
 TCP window size: 43.8 KByte (default)
 ------------------------------------------------------------
 [  3] local 10.42.0.243 port 60248 connected with 10.42.0.1 port 5001
 [ ID] Interval       Transfer     Bandwidth
 [  3]  0.0-10.0 sec  47.5 MBytes  39.8 Mbits/sec
 # iperf -c 10.42.0.1
 ------------------------------------------------------------
 Client connecting to 10.42.0.1, TCP port 5001
 TCP window size: 43.8 KByte (default)
 ------------------------------------------------------------
 [  3] local 10.42.0.243 port 60249 connected with 10.42.0.1 port 5001
 [ ID] Interval       Transfer     Bandwidth
 [  3]  0.0-10.0 sec  54.9 MBytes  46.0 Mbits/sec
 ============================================================
Signed-off-by: NAlexey Brodkin <abrodkin@synopsys.com>
Cc: arc-linux-dev@synopsys.com
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

L2 cache on ARCHS processors is called SLC (System Level Cache)
For working DMA (in absence of hardware assisted IO Coherency) we need
to manage SLC explicitly when buffers transition between cpu and
controllers.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

This is also default for AXS103 release
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Caveats about cache flush on ARCv2 based cores

- dcache is PIPT so paddr is sufficient for cache maintenance ops (no
  need to setup PTAG reg

- icache is still VIPT but only aliasing configs need PTAG setup

So basically this is departure from MMU-v3 which always need vaddr in
line ops registers (DC_IVDL, DC_FLDL, IC_IVIL) but paddr in DC_PTAG,
IC_PTAG respectively.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

- Remove the ifdef'ery and write distinct versions for each mmu ver even
  if there is some code duplication
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

That is because __after_dc_op() already reads it for status check, so it
is better anyways to use that "newer" value.

Also reduces the clutter in callers for passing from/to these routines.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Structure name and variable name were erroneously interchanged
Signed-off-by: NNoam Camus <noamc@ezchip.com>
Acked-by: NVineet Gupta <vgupta@synopsys.com>
[ Also removed pointless cast from "void *".  - Linus ]
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Fix building errors occuring due to a missing export of
flush_icache_range() in

kisskb.ellerman.id.au/kisskb/buildresult/11677809/

ERROR: "flush_icache_range" [drivers/misc/lkdtm.ko] undefined!
Signed-off-by: NPranith Kumar <bobby.prani@gmail.com>
Reported-by: NGeert Uytterhoeven <geert@linux-m68k.org>
Acked-by: Vineet Gupta <vgupta@synopsys.com>	[arc]
Acked-by: Richard Kuo <rkuo@codeaurora.org>	[hexagon]
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Chris Zankel <chris@zankel.net>
Acked-by: Max Filippov <jcmvbkbc@gmail.com>	[xtensa]
Cc: Noam Camus <noamc@ezchip.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Acked-by: Zhigang Lu <zlu@tilera.com>		[tile]
Cc: Kirill Tkhai <tkhai@yandex.ru>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

INV cmd for dcache provides 2 modes discard or wback-before-discard.
One is default and other needs to be set, if so desired. This is common
for line-op/entire-cache-op. So refactor them out into a helper

Doesn't affect generated code but paves way for any common
micro-optimization.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

* print aliasing or not, VIPT/PIPT etc
* compress param storage using bitfields
* more use of IS_ENABLED to de-uglify code
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

icaches are not snooped hence not cohrent in SMP setups which means
kernel has to do cross core calls to ensure the same.

The leaf routine __ic_line_inv_vaddr() now does cross core calls.

__sync_icache_dcache() is affected due to this:

* local dcache line flushed ahead of remote icache inv requests
* can't disable interrupts anymore, since
      __ic_line_inv_vaddr()->on_each_cpu() can deadlock.

| WARNING: CPU: 0 PID: 1 at kernel/smp.c:374
| smp_call_function_many+0x25a/0x2c4()
|
|  init_kprobes+0x90/0xc8
|     register_kprobe+0x1d6/0x510
|	__sync_icache_dcache+0x28/0x80
|
|	    DISABLE IRQ
|
|	    __ic_line_inv_vaddr
|		on_each_cpu
|		     smp_call_function_many+0x25a/0x2c4   --> WARN
|			__ic_line_inv_vaddr_local
|	    __dc_line_op

* TODO: Needs to use mask of relevant CPUs to avoid broadcasting
Signed-off-by: NNoam Camus <noamc@ezchip.com>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Requested-by: NNoam Camus <noamc@ezchip.com>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

-Pass the expected arg to non-boot park'ing routine
 (It worked so far because existing SMP backends don't use the arg)

-CONFIG_DEBUG_PREEMPT warning

This fixes a subtle issue with cache flush which could potentially cause
random userspace crashes because of stale icache lines.

This error crept in when consolidating the cache flush code

Fixes: bd12976c (ARC: cacheflush refactor #3: Unify the {d,i}cache)
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>
Cc: linux-kernel@vger.kernel.org
Cc: stable@vger.kernel.org  # 3.13
Cc: arc-linux-dev@synopsys.com
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Line op needs vaddr (indexing) and paddr (tag match). For page sized
flushes (V-P const), each line op will need a different index, but the
tag bits wil remain constant, hence paddr can be setup once outside the
loop.

This improves select LMBench numbers for Aliasing dcache where we have
more "preventive" cache flushing.

Processor, Processes - times in microseconds - smaller is better
------------------------------------------------------------------------------
Host                 OS  Mhz null null      open slct sig  sig  fork exec sh
                             call  I/O stat clos TCP  inst hndl proc proc proc
--------- ------------- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
3.11-rc7- Linux 3.11.0-   80 4.66 8.88 69.7 112. 268. 8.60 28.0 3489 13.K 27.K	# Non alias ARC700
3.11-rc7- Linux 3.11.0-   80 4.64 8.51 68.6 98.5 271. 8.58 28.1 4160 15.K 32.K	# Aliasing
3.11-rc7- Linux 3.11.0-   80 4.64 8.51 69.8 99.4 270. 8.73 27.5 3880 15.K 31.K	# PTAG loop Inv
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

With Line length being constant now, we can fold the 2 helpers into 1.
This allows applying any optimizations (forthcoming) to single place.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Having them be different seems an obscure configuration.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

ARC dcache supports 3 ops - Inv, Flush, Flush-n-Inv.
The programming model however provides 2 commands FLUSH, INV.
INV will either discard or flush-n-discard (based on DT_CTRL bit)

The leaf helper __dc_line_loop() used to take the AUX register
(corresponding to the 2 commands). Now we push that to within the
helper, paving way for code consolidations to follow.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

* reduce editor lines taken by pt_regs
* ARCompact ISA specific part of TLB Miss handlers clubbed together
* cleanup some comments
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

The __cpuinit type of throwaway sections might have made sense
some time ago when RAM was more constrained, but now the savings
do not offset the cost and complications.  For example, the fix in
commit 5e427ec2 ("x86: Fix bit corruption at CPU resume time")
is a good example of the nasty type of bugs that can be created
with improper use of the various __init prefixes.

After a discussion on LKML[1] it was decided that cpuinit should go
the way of devinit and be phased out.  Once all the users are gone,
we can then finally remove the macros themselves from linux/init.h.

Note that some harmless section mismatch warnings may result, since
notify_cpu_starting() and cpu_up() are arch independent (kernel/cpu.c)
are flagged as __cpuinit  -- so if we remove the __cpuinit from
arch specific callers, we will also get section mismatch warnings.
As an intermediate step, we intend to turn the linux/init.h cpuinit
content into no-ops as early as possible, since that will get rid
of these warnings.  In any case, they are temporary and harmless.

This removes all the arch/arc uses of the __cpuinit macros from
all C files.  Currently arc does not have any __CPUINIT used in
assembly files.

[1] https://lkml.org/lkml/2013/5/20/589

Cc: Vineet Gupta <vgupta@synopsys.com>
Signed-off-by: NPaul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Similar to ARM/SH
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Non-congruent SRC page in copy_user_page() is dcache clean in the end -
so record that fact, to avoid a subsequent extraneous flush.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

flush_cache_page() - kills icache only if page is executable
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Also remove extraneous irq disabling in flush_cache_all() callstack
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

* Number of (i|d)cache ways can be retrieved from BCRs and hence no need
  to cross check with with built-in constants
* Use of IS_ENABLED() to check for a Kconfig option
* is_not_cache_aligned() not used anymore
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

* Move the various sub-system defines/types into relevant files/functions
  (reduces compilation time)

* move CPU specific stuff out of asm/tlb.h into asm/mmu.h
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>