Signed-off-by: NAndrea Gelmini <andrea.gelmini@gelma.net>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.

This promise never materialized.  And unlikely will.

We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE.  And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.

Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.

Let's stop pretending that pages in page cache are special.  They are
not.

The changes are pretty straight-forward:

 - <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;

 - <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;

 - PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};

 - page_cache_get() -> get_page();

 - page_cache_release() -> put_page();

This patch contains automated changes generated with coccinelle using
script below.  For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.

The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.

There are few places in the code where coccinelle didn't reach.  I'll
fix them manually in a separate patch.  Comments and documentation also
will be addressed with the separate patch.

virtual patch

@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E

@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E

@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT

@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE

@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK

@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)

@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)

@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: NKirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: NMichal Hocko <mhocko@suse.com>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The peripheral address space is architectural address window which is
uncached and typically used to wire up peripherals.

For ARC700 cores (ARCompact ISA based) this was fixed to 1GB region
0xC000_0000 - 0xFFFF_FFFF.

For ARCv2 based HS38 cores the start address is flexible and can be
0xC, 0xD, 0xE, 0xF 000_000 by programming AUX_NON_VOLATILE_LIMIT reg
(typically done in bootloader)

Further in cas of PAE, the physical address can extend beyond 4GB so
need to confine this check, otherwise all pages beyond 4GB will be
treated as uncached
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

To support dma in physical memory beyond 4GB with PAE40
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NAdam Buchbinder <adam.buchbinder@gmail.com>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Let's define page_mapped() to be true for compound pages if any
sub-pages of the compound page is mapped (with PMD or PTE).

On other hand page_mapcount() return mapcount for this particular small
page.

This will make cases like page_get_anon_vma() behave correctly once we
allow huge pages to be mapped with PTE.

Most users outside core-mm should use page_mapcount() instead of
page_mapped().
Signed-off-by: NKirill A. Shutemov <kirill.shutemov@linux.intel.com>
Tested-by: NSasha Levin <sasha.levin@oracle.com>
Tested-by: NAneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Acked-by: NJerome Marchand <jmarchan@redhat.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Rik van Riel <riel@redhat.com>
Cc: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
Cc: Steve Capper <steve.capper@linaro.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: Christoph Lameter <cl@linux.com>
Cc: David Rientjes <rientjes@google.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

This is the first working implementation of 40-bit physical address
extension on ARCv2.
Signed-off-by: NAlexey Brodkin <abrodkin@synopsys.com>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

That way a single flip of phys_addr_t to 64 bit ensures all places
dealing with physical addresses get correct data
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Before we plug in highmem support, some of code needs to be ready for it
 - copy_user_highpage() needs to be using the kmap_atomic API
 - mk_pte() can't assume page_address()
 - do_page_fault() can't assume VMALLOC_END is end of kernel vaddr space
Signed-off-by: NAlexey Brodkin <abrodkin@synopsys.com>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NAlexey Brodkin <abrodkin@synopsys.com>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

In case of ARCv2 CPU there're could be following configurations
that affect cache handling for data exchanged with peripherals
via DMA:
 [1] Only L1 cache exists
 [2] Both L1 and L2 exist, but no IO coherency unit
 [3] L1, L2 caches and IO coherency unit exist

Current implementation takes care of [1] and [2].
Moreover support of [2] is implemented with run-time check
for SLC existence which is not super optimal.

This patch introduces support of [3] and rework of DMA ops
usage. Instead of doing run-time check every time a particular
DMA op is executed we'll have 3 different implementations of
DMA ops and select appropriate one during init.

As for IOC support for it we need:
 [a] Implement empty DMA ops because IOC takes care of cache
     coherency with DMAed data
 [b] Route dma_alloc_coherent() via dma_alloc_noncoherent()
     This is required to make IOC work in first place and also
     serves as optimization as LD/ST to coherent buffers can be
     srviced from caches w/o going all the way to memory
Signed-off-by: NAlexey Brodkin <abrodkin@synopsys.com>
[vgupta:
  -Added some comments about IOC gains
  -Marked dma ops as static,
  -Massaged changelog a bit]
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

SLC maintenance ops need to be serialized by software as there is no
inherent buffering / quequing of aux commands. It can silently ignore a
new aux operation if previous one is still ongoing (SLC_CTRL_BUSY)

So gaurd the SLC op using a spin lock

The spin lock doesn't seem to be contended even in heavy workloads such
as iperf. On FPGA @ 75 MHz.

 [1] Before this change:
 ============================================================
  # iperf -c 10.42.0.1
 ------------------------------------------------------------
 Client connecting to 10.42.0.1, TCP port 5001
 TCP window size: 43.8 KByte (default)
 ------------------------------------------------------------
 [  3] local 10.42.0.110 port 38935 connected with 10.42.0.1 port 5001
 [ ID] Interval       Transfer     Bandwidth
 [  3]  0.0-10.0 sec  48.4 MBytes  40.6 Mbits/sec
 ============================================================

 [2] After this change:
 ============================================================
 # iperf -c 10.42.0.1
 ------------------------------------------------------------
 Client connecting to 10.42.0.1, TCP port 5001
 TCP window size: 43.8 KByte (default)
 ------------------------------------------------------------
 [  3] local 10.42.0.243 port 60248 connected with 10.42.0.1 port 5001
 [ ID] Interval       Transfer     Bandwidth
 [  3]  0.0-10.0 sec  47.5 MBytes  39.8 Mbits/sec
 # iperf -c 10.42.0.1
 ------------------------------------------------------------
 Client connecting to 10.42.0.1, TCP port 5001
 TCP window size: 43.8 KByte (default)
 ------------------------------------------------------------
 [  3] local 10.42.0.243 port 60249 connected with 10.42.0.1 port 5001
 [ ID] Interval       Transfer     Bandwidth
 [  3]  0.0-10.0 sec  54.9 MBytes  46.0 Mbits/sec
 ============================================================
Signed-off-by: NAlexey Brodkin <abrodkin@synopsys.com>
Cc: arc-linux-dev@synopsys.com
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

L2 cache on ARCHS processors is called SLC (System Level Cache)
For working DMA (in absence of hardware assisted IO Coherency) we need
to manage SLC explicitly when buffers transition between cpu and
controllers.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

This is also default for AXS103 release
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Caveats about cache flush on ARCv2 based cores

- dcache is PIPT so paddr is sufficient for cache maintenance ops (no
  need to setup PTAG reg

- icache is still VIPT but only aliasing configs need PTAG setup

So basically this is departure from MMU-v3 which always need vaddr in
line ops registers (DC_IVDL, DC_FLDL, IC_IVIL) but paddr in DC_PTAG,
IC_PTAG respectively.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

- Remove the ifdef'ery and write distinct versions for each mmu ver even
  if there is some code duplication
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

That is because __after_dc_op() already reads it for status check, so it
is better anyways to use that "newer" value.

Also reduces the clutter in callers for passing from/to these routines.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Structure name and variable name were erroneously interchanged
Signed-off-by: NNoam Camus <noamc@ezchip.com>
Acked-by: NVineet Gupta <vgupta@synopsys.com>
[ Also removed pointless cast from "void *".  - Linus ]
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Fix building errors occuring due to a missing export of
flush_icache_range() in

kisskb.ellerman.id.au/kisskb/buildresult/11677809/

ERROR: "flush_icache_range" [drivers/misc/lkdtm.ko] undefined!
Signed-off-by: NPranith Kumar <bobby.prani@gmail.com>
Reported-by: NGeert Uytterhoeven <geert@linux-m68k.org>
Acked-by: Vineet Gupta <vgupta@synopsys.com>	[arc]
Acked-by: Richard Kuo <rkuo@codeaurora.org>	[hexagon]
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Chris Zankel <chris@zankel.net>
Acked-by: Max Filippov <jcmvbkbc@gmail.com>	[xtensa]
Cc: Noam Camus <noamc@ezchip.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Acked-by: Zhigang Lu <zlu@tilera.com>		[tile]
Cc: Kirill Tkhai <tkhai@yandex.ru>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

INV cmd for dcache provides 2 modes discard or wback-before-discard.
One is default and other needs to be set, if so desired. This is common
for line-op/entire-cache-op. So refactor them out into a helper

Doesn't affect generated code but paves way for any common
micro-optimization.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

* print aliasing or not, VIPT/PIPT etc
* compress param storage using bitfields
* more use of IS_ENABLED to de-uglify code
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

icaches are not snooped hence not cohrent in SMP setups which means
kernel has to do cross core calls to ensure the same.

The leaf routine __ic_line_inv_vaddr() now does cross core calls.

__sync_icache_dcache() is affected due to this:

* local dcache line flushed ahead of remote icache inv requests
* can't disable interrupts anymore, since
      __ic_line_inv_vaddr()->on_each_cpu() can deadlock.

| WARNING: CPU: 0 PID: 1 at kernel/smp.c:374
| smp_call_function_many+0x25a/0x2c4()
|
|  init_kprobes+0x90/0xc8
|     register_kprobe+0x1d6/0x510
|	__sync_icache_dcache+0x28/0x80
|
|	    DISABLE IRQ
|
|	    __ic_line_inv_vaddr
|		on_each_cpu
|		     smp_call_function_many+0x25a/0x2c4   --> WARN
|			__ic_line_inv_vaddr_local
|	    __dc_line_op

* TODO: Needs to use mask of relevant CPUs to avoid broadcasting
Signed-off-by: NNoam Camus <noamc@ezchip.com>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Requested-by: NNoam Camus <noamc@ezchip.com>
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

-Pass the expected arg to non-boot park'ing routine
 (It worked so far because existing SMP backends don't use the arg)

-CONFIG_DEBUG_PREEMPT warning

This fixes a subtle issue with cache flush which could potentially cause
random userspace crashes because of stale icache lines.

This error crept in when consolidating the cache flush code

Fixes: bd12976c (ARC: cacheflush refactor #3: Unify the {d,i}cache)
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>
Cc: linux-kernel@vger.kernel.org
Cc: stable@vger.kernel.org  # 3.13
Cc: arc-linux-dev@synopsys.com
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Line op needs vaddr (indexing) and paddr (tag match). For page sized
flushes (V-P const), each line op will need a different index, but the
tag bits wil remain constant, hence paddr can be setup once outside the
loop.

This improves select LMBench numbers for Aliasing dcache where we have
more "preventive" cache flushing.

Processor, Processes - times in microseconds - smaller is better
------------------------------------------------------------------------------
Host                 OS  Mhz null null      open slct sig  sig  fork exec sh
                             call  I/O stat clos TCP  inst hndl proc proc proc
--------- ------------- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
3.11-rc7- Linux 3.11.0-   80 4.66 8.88 69.7 112. 268. 8.60 28.0 3489 13.K 27.K	# Non alias ARC700
3.11-rc7- Linux 3.11.0-   80 4.64 8.51 68.6 98.5 271. 8.58 28.1 4160 15.K 32.K	# Aliasing
3.11-rc7- Linux 3.11.0-   80 4.64 8.51 69.8 99.4 270. 8.73 27.5 3880 15.K 31.K	# PTAG loop Inv
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

With Line length being constant now, we can fold the 2 helpers into 1.
This allows applying any optimizations (forthcoming) to single place.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

Having them be different seems an obscure configuration.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

ARC dcache supports 3 ops - Inv, Flush, Flush-n-Inv.
The programming model however provides 2 commands FLUSH, INV.
INV will either discard or flush-n-discard (based on DT_CTRL bit)

The leaf helper __dc_line_loop() used to take the AUX register
(corresponding to the 2 commands). Now we push that to within the
helper, paving way for code consolidations to follow.
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>

* reduce editor lines taken by pt_regs
* ARCompact ISA specific part of TLB Miss handlers clubbed together
* cleanup some comments
Signed-off-by: NVineet Gupta <vgupta@synopsys.com>