The check cpu_needs_post_dma_flush() in mips_dma_sync_sg_for_cpu() and
the check !plat_device_is_coherent() in mips_dma_sync_sg_for_device()
can be moved outside the for loop.

As a side effect, this also avoids a GCC bug that caused kernel compile
to fail with the error:

arch/mips/mm/dma-default.c: In function 'mips_dma_sync_sg_for_cpu':
arch/mips/mm/dma-default.c:316:1: internal compiler error: in add_insn_before, at emit-rtl.c:3852

This gcc failure is seen in Code Sourcery toolchains [e.g. gcc version
4.7.2 (Sourcery CodeBench Lite 2012.09-99)] after commit "MIPS: Optimize
current_cpu_type() for better code."
Signed-off-by: NJayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5907/Reviewed-by: NMarkos Chandras <markos.chandras@imgtec.com>
Tested-by: NMarkos Chandras <markos.chandras@imgtec.com>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

 o Move current_cpu_type() to a separate header file
 o #ifdefing on supported CPU types lets modern GCC know that certain
   code in callers may be discarded ideally turning current_cpu_type() into
   a function returning a constant.
 o Use current_cpu_type() rather than direct access to struct cpuinfo_mips.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5833/

The use of current_cpu_type() in cpu_is_noncoherent_r10000() is not preemption-safe.
Use boot_cpu_type() instead to make it preemption-safe.

<log>
/ # insmod mtd_readtest.ko dev=4
mtd_readtest: MTD device: 4
mtd_readtest: MTD device size 996671488, eraseblock size 524288, page size 4096, count of eraseblocks 1901, pages per eraseblock 128, OOB size 224
mtd_readtest: scanning for bad eraseblocks
mtd_readtest: scanned 1901 eraseblocks, 0 are bad
mtd_readtest: testing page read
BUG: using smp_processor_id() in preemptible [00000000] code: insmod/99
caller is mips_dma_sync_single_for_cpu+0x2c/0x128
CPU: 2 PID: 99 Comm: insmod Not tainted 3.10.4 #67
Stack : 00000006 69735f63 00000000 00000000 00000000 00000000 808273d6 00000032
          80820000 00000002 8d700000 8de48fa0 00000000 00000000 00000000 00000000
          00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
          00000000 00000000 00000000 8d6afb00 8d6afb24 80721f24 807b9927 8012c130
          80820000 80721f24 00000002 00000063 8de48fa0 8082333c 807b98e6 8d6afaa0
          ...
Call Trace:
[<80109984>] show_stack+0x64/0x7c
[<80666230>] dump_stack+0x20/0x2c
[<803a2210>] debug_smp_processor_id+0xe0/0xf0
[<801116f0>] mips_dma_sync_single_for_cpu+0x2c/0x128
[<8043456c>] nand_plat_read_page+0x16c/0x234
[<8042fad4>] nand_do_read_ops+0x194/0x480
[<804301dc>] nand_read+0x50/0x7c
[<804261c8>] part_read+0x70/0xc0
[<804231dc>] mtd_read+0x80/0xe4
[<c0431354>] init_module+0x354/0x6f8 [mtd_readtest]
[<8010057c>] do_one_initcall+0x140/0x1a4
[<80176d7c>] load_module+0x1b5c/0x2258
[<8017752c>] SyS_init_module+0xb4/0xec
[<8010f3fc>] stack_done+0x20/0x44

BUG: using smp_processor_id() in preemptible [00000000] code: insmod/99
</log>
Signed-off-by: NJerin Jacob <jerinjacobk@gmail.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5800/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

The BMIPS5000 (Zephyr) processor utilizes instruction speculation. A
stale misprediction address in either the JTB or the CRS may trigger
a prefetch inside a region that is currently being used by a DMA engine,
which is not IO-coherent.  This prefetch will fetch a line into the
scache, and that line will soon become stale (ie wrong) during/after the
DMA.  Mayhem ensues.

In dma-default.c, the r10000 is handled as a special case in the same way
that we want to handle Zephyr.  So we generalize the exception cases into
a function, and include Zephyr as one of the processors that needs this
special care.
Signed-off-by: NJim Quinlan <jim2101024@gmail.com>
Cc: linux-mips@linux-mips.org
Cc: cernekee@gmail.com
Patchwork: https://patchwork.linux-mips.org/patch/5776/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Provide a default implementation of phys_to_dma and dma_to_phys in
mach-generic/dma_coherence.h.

If CONFIG_NEED_SG_DMA_LENGTH is defined, the dma_length field in
struct scatterlist is used. Set this up in mips_dma_map_sg so that
the default mips DMA ops can be used when SWIOTLB is enabled.
Signed-off-by: NJayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5409/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Some MIPS controllers have hardware I/O coherency. This patch
detects those and turns off software coherency. A new kernel
command line option also allows the user to manually turn
software coherency on or off.
Signed-off-by: NSteven J. Hill <Steven.Hill@imgtec.com>

Having received another series of whitespace patches I decided to do this
once and for all rather than dealing with this kind of patches trickling
in forever.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Adapt core MIPS architecture code for dma_map_ops changes: replace
alloc/free_coherent with generic alloc/free methods.
Signed-off-by: NAndrzej Pietrasiewicz <andrzej.p@samsung.com>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
[added missing changes to arch/mips/cavium-octeon/dma-octeon.c,
 fixed attrs argument in dma-mapping.h]
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Reviewed-by: NArnd Bergmann <arnd@arndb.de>

[v4: Patch applies to linux-queue.git with kmap_atomic patches:
 https://patchwork.kernel.org/patch/189932/
 https://patchwork.kernel.org/patch/194552/
 https://patchwork.kernel.org/patch/189912/ ]

The MIPS DMA coherency functions do not work properly (i.e. kernel oops)
when HIGHMEM pages are passed in as arguments.  Use kmap_atomic() to
temporarily map high pages for cache maintenance operations.

Tested on a 2.6.36-rc7 1GB HIGHMEM SMP no-alias system.
Signed-off-by: NDezhong Diao <dediao@cisco.com>
Signed-off-by: NKevin Cernekee <cernekee@gmail.com>
Cc: Dezhong Diao <dediao@cisco.com>
Cc: David Daney <ddaney@caviumnetworks.com>
Cc: David VomLehn <dvomlehn@cisco.com>
Cc: Sergei Shtylyov <sshtylyov@mvista.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/1695/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

This fixes IP22 and IP28 build errors.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Use asm-generic/dma-mapping-common.h to handle all DMA mapping operations
and establish a default get_dma_ops() that forwards all operations to the
existing code.

Augment dev_archdata to carry a pointer to the struct dma_map_ops, allowing
DMA operations to be overridden on a per device basis.  Currently this is
never filled in, so the default dma_map_ops are used.  A follow-on patch
sets this for Octeon PCI devices.

Also initialize the dma_debug system as it is now used if it is configured.

Includes fixes by Kevin Cernekee <cernekee@gmail.com>.
Signed-off-by: NDavid Daney <ddaney@caviumnetworks.com>
Patchwork: http://patchwork.linux-mips.org/patch/1637/
Patchwork: http://patchwork.linux-mips.org/patch/1678/Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

This only matters for ISA devices with a 24-bit DMA limit or for devices
with a 32-bit DMA limit on systems with ZONE_DMA32 enabled.  The latter
currently only affects 32-bit PCI cards on Sibyte-based systems with more
than 1GB RAM installed.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Architectures implement dma_is_consistent() in different ways (some
misinterpret the definition of API in DMA-API.txt).  So it hasn't been so
useful for drivers.  We have only one user of the API in tree.  Unlikely
out-of-tree drivers use the API.

Even if we fix dma_is_consistent() in some architectures, it doesn't look
useful at all.  It was invented long ago for some old systems that can't
allocate coherent memory at all.  It's better to export only APIs that are
definitely necessary for drivers.

Let's remove this API.
Signed-off-by: NFUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Cc: James Bottomley <James.Bottomley@HansenPartnership.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: <linux-arch@vger.kernel.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h

percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files.  percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.

percpu.h -> slab.h dependency is about to be removed.  Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability.  As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.

  http://userweb.kernel.org/~tj/misc/slabh-sweep.py

The script does the followings.

* Scan files for gfp and slab usages and update includes such that
  only the necessary includes are there.  ie. if only gfp is used,
  gfp.h, if slab is used, slab.h.

* When the script inserts a new include, it looks at the include
  blocks and try to put the new include such that its order conforms
  to its surrounding.  It's put in the include block which contains
  core kernel includes, in the same order that the rest are ordered -
  alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
  doesn't seem to be any matching order.

* If the script can't find a place to put a new include (mostly
  because the file doesn't have fitting include block), it prints out
  an error message indicating which .h file needs to be added to the
  file.

The conversion was done in the following steps.

1. The initial automatic conversion of all .c files updated slightly
   over 4000 files, deleting around 700 includes and adding ~480 gfp.h
   and ~3000 slab.h inclusions.  The script emitted errors for ~400
   files.

2. Each error was manually checked.  Some didn't need the inclusion,
   some needed manual addition while adding it to implementation .h or
   embedding .c file was more appropriate for others.  This step added
   inclusions to around 150 files.

3. The script was run again and the output was compared to the edits
   from #2 to make sure no file was left behind.

4. Several build tests were done and a couple of problems were fixed.
   e.g. lib/decompress_*.c used malloc/free() wrappers around slab
   APIs requiring slab.h to be added manually.

5. The script was run on all .h files but without automatically
   editing them as sprinkling gfp.h and slab.h inclusions around .h
   files could easily lead to inclusion dependency hell.  Most gfp.h
   inclusion directives were ignored as stuff from gfp.h was usually
   wildly available and often used in preprocessor macros.  Each
   slab.h inclusion directive was examined and added manually as
   necessary.

6. percpu.h was updated not to include slab.h.

7. Build test were done on the following configurations and failures
   were fixed.  CONFIG_GCOV_KERNEL was turned off for all tests (as my
   distributed build env didn't work with gcov compiles) and a few
   more options had to be turned off depending on archs to make things
   build (like ipr on powerpc/64 which failed due to missing writeq).

   * x86 and x86_64 UP and SMP allmodconfig and a custom test config.
   * powerpc and powerpc64 SMP allmodconfig
   * sparc and sparc64 SMP allmodconfig
   * ia64 SMP allmodconfig
   * s390 SMP allmodconfig
   * alpha SMP allmodconfig
   * um on x86_64 SMP allmodconfig

8. percpu.h modifications were reverted so that it could be applied as
   a separate patch and serve as bisection point.

Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: NTejun Heo <tj@kernel.org>
Guess-its-ok-by: NChristoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>

The ohci-sm501 driver requires dma_declare_coherent_memory().  It is used
by the driver's local memory allocation with dma_alloc_coherent().

Tested on TANBAC TB0287(VR4131 + SM501).

[Ralf: Fixed reject in dma-default.c and removed the entire #if 0'ed block
 in dma-mapping.h instead of just the #if 0.]
Signed-off-by: NYoichi Yuasa <yoichi_yuasa@tripeaks.co.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NKevin Cernekee <cernekee@gmail.com>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NKevin Cernekee <cernekee@gmail.com>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Synchronize dma_map_page/dma_unmap_page and dma_map_single/dma_unmap_single.
This will reduce unnecessary writebacks and invalidates.

[Ralf: make dma_unmap_page an inline function.]
Signed-off-by: NAtsushi Nemoto <anemo@mba.ocn.ne.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

dma_cache_wback_inv() expects virtual address, but physical was provided
due to translation via plat_dma_addr_to_phys().
If replaced with dma_addr_to_virt(), page fault oops from dma_unmap_page()
is gone on au1550 platform.
Signed-off-by: NJan Nikitenko <jan.nikitenko@gmail.com>
Acked-by: NAtsushi Nemoto <anemo@mba.ocn.ne.jp>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

We add a dev parameter to plat_unmap_dma_mem(), and hooks for
plat_dma_supported() and plat_extra_sync_for_device() which should be
nop changes for all existing targets.
Signed-off-by: NDavid Daney <ddaney@caviumnetworks.com>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

dma_free_noncoherent() and dma_free_coherent() are missing calls to
plat_unmap_dma_mem().  This patch adds them.
Signed-off-by: NDavid Daney <ddaney@caviumnetworks.com>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

We were getting away with this for so long only because the only platform
with a non-empty plat_unmap_dma_mem() doesn't call dma_sync_sg_for_cpu()
and dma_sync_sg_for_device() from its commonly used drivers.
Signed-off-by: NDavid Daney <ddaney@caviumnetworks.com>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Add per-device dma_mapping_ops support for CONFIG_X86_64 as POWER
architecture does:

This enables us to cleanly fix the Calgary IOMMU issue that some devices
are not behind the IOMMU (http://lkml.org/lkml/2008/5/8/423).

I think that per-device dma_mapping_ops support would be also helpful for
KVM people to support PCI passthrough but Andi thinks that this makes it
difficult to support the PCI passthrough (see the above thread).  So I
CC'ed this to KVM camp.  Comments are appreciated.

A pointer to dma_mapping_ops to struct dev_archdata is added.  If the
pointer is non NULL, DMA operations in asm/dma-mapping.h use it.  If it's
NULL, the system-wide dma_ops pointer is used as before.

If it's useful for KVM people, I plan to implement a mechanism to register
a hook called when a new pci (or dma capable) device is created (it works
with hot plugging).  It enables IOMMUs to set up an appropriate
dma_mapping_ops per device.

The major obstacle is that dma_mapping_error doesn't take a pointer to the
device unlike other DMA operations.  So x86 can't have dma_mapping_ops per
device.  Note all the POWER IOMMUs use the same dma_mapping_error function
so this is not a problem for POWER but x86 IOMMUs use different
dma_mapping_error functions.

The first patch adds the device argument to dma_mapping_error.  The patch
is trivial but large since it touches lots of drivers and dma-mapping.h in
all the architecture.

This patch:

dma_mapping_error() doesn't take a pointer to the device unlike other DMA
operations.  So we can't have dma_mapping_ops per device.

Note that POWER already has dma_mapping_ops per device but all the POWER
IOMMUs use the same dma_mapping_error function.  x86 IOMMUs use device
argument.

[akpm@linux-foundation.org: fix sge]
[akpm@linux-foundation.org: fix svc_rdma]
[akpm@linux-foundation.org: build fix]
[akpm@linux-foundation.org: fix bnx2x]
[akpm@linux-foundation.org: fix s2io]
[akpm@linux-foundation.org: fix pasemi_mac]
[akpm@linux-foundation.org: fix sdhci]
[akpm@linux-foundation.org: build fix]
[akpm@linux-foundation.org: fix sparc]
[akpm@linux-foundation.org: fix ibmvscsi]
Signed-off-by: NFUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Cc: Muli Ben-Yehuda <muli@il.ibm.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Avi Kivity <avi@qumranet.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Not cache coherent R10k systems (like IP28) need to do real cache
invalidates in dma_cache_sync().
Signed-off-by: NThomas Bogendoerfer <tsbogend@alpha.franken.de>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NThomas Bogendoerfer <tsbogend@alpha.franken.de>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Sibyte SOCs only have 32-bit PCI.  Due to the sparse use of the address
space only the first 1GB of memory is mapped at physical addresses
below 1GB.  If a system has more than 1GB of memory 32-bit DMA will
not be able to reach all of it.

For now this patch is good enough to keep Sibyte users happy but it seems
eventually something like swiotlb will be needed for Sibyte.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NJens Axboe <jens.axboe@oracle.com>

Signed-off-by: NJens Axboe <jens.axboe@oracle.com>

This saves a few k on systems which only ever ship with a single CPU type.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NMaxime Bizon <mbizon@freebox.fr>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NMaxime Bizon <mbizon@freebox.fr>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NThomas Bogendoerfer <tsbogend@alpha.franken.de>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

This patch fixes two places where we used plain 'x - PAGE_OFFSET' to
achieve virtual to physical address convertions. This type of convertion
is no more allowed since commit 6f284a2c.
Reported-by: NMaxime Bizon <mbizon@freebox.fr>
Signed-off-by: NFranck Bui-Huu <fbuihuu@gmail.com>

[Build fixes for machines that don't use the generic dma-coherence.h]
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Signed-off-by: NThomas Bogendoerfer <tsbogend@alpha.franken.de>
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Platforms will now have to supply a function dma_device_is_coherent which
returns if a particular device participates in the coherence domain.  For
most platforms this function will always return 0 or 1.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

Pass struct dev pointer to dma_cache_sync()

dma_cache_sync() is ill-designed in that it does not have a struct device
pointer argument which makes proper support for systems that consist of a
mix of coherent and non-coherent DMA devices hard.  Change dma_cache_sync
to take a struct device pointer as first argument and fix all its callers
to pass it.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>
Cc: James Bottomley <James.Bottomley@steeleye.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Greg KH <greg@kroah.com>
Signed-off-by: NAndrew Morton <akpm@osdl.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

dma_is_consistent() is ill-designed in that it does not have a struct
device pointer argument which makes proper support for systems that consist
of a mix of coherent and non-coherent DMA devices hard.  Change
dma_is_consistent to take a struct device pointer as first argument and fix
the sole caller to pass it.
Signed-off-by: NRalf Baechle <ralf@linux-mips.org>
Cc: James Bottomley <James.Bottomley@steeleye.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Greg KH <greg@kroah.com>
Signed-off-by: NAndrew Morton <akpm@osdl.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Signed-off-by: NJörn Engel <joern@wohnheim.fh-wedel.de>
Signed-off-by: NAdrian Bunk <bunk@stusta.de>