Move all DMA descriptor prepping functions to prep.c file. Fixup all
broken bits caused by the move.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Moving all the init routines to init.c and fixup anything broken during
the move.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Move and fixup all sysfs related bits to sysfs.c file.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Clean out dma_v2 and remove ioat2 calls since we are moving everything
to just ioat.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Changing the variable names for ioatdma_device to be consistently named
ioat_dma instead of device/dma in order to avoid confusion and distinct
from struct device. This will clearly indicate that it is an
ioatdma_device. This also make all the naming consistent that the dma
device is ioat_dma and all the channels are ioat_chan.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Kill the common ioatdma channel structure and everything that is not
dma_chan to be ioat_dma_chan. Since we don't have to worry about v1
and v2 ioatdma anymore this makes it much cleaner and obvious for
maintenance.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Removal of support for ioatdma v2 device support.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Cleaning up of ioat1 specific code as it is no longer supported
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Removal of any devices that are ioatdma pre-3.0. This is the first step
in attempting to clean up the ioatdma driver and remove hw no longer
supported.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If the allocation order is 16, then the u16 count will overflow and wrap
to zero when assigned the value 1 << 16.

Change the type of 'total_descs' to int, so that it is large enough to
store a value equal or greater than 1 << 16.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Acked-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If the allocation order is 16, then the u16 index will overflow and wrap
to zero instead of being equal or greater than 1 << 16.  The loop
condition will always be true, and the loop will run until all the
memory resources are depleted.

Change the type of index 'i' to u32, so that it is large enough to store
a value equal or greater than 1 << 16.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Acked-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The functions irq_irq_err and ipu_irq_fn are identical plus/minus the
comments. Remove one.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: dmaengine@vger.kernel.org
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The irq argument of most interrupt flow handlers is unused or merily
used instead of a local variable. The handlers which need the irq
argument can retrieve the irq number from the irq descriptor.

Search and update was done with coccinelle and the invaluable help of
Julia Lawall.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Cc: Julia Lawall <Julia.Lawall@lip6.fr>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: dmaengine@vger.kernel.org
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The XDMAC also supports memset operations over discontiguous areas. Add the
necessary logic to support this.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NLudovic Desroches <ludovic.desroches@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Most drivers need to set constraints on the buffer alignment for async tx
operations. However, even though it is documented, some drivers either use
a defined constant that is not matching what the alignment variable expects
(like DMA_BUSWIDTH_* constants) or fill the alignment in bytes instead of
power of two.

Add a new enum for these alignments that matches what the framework
expects, and convert the drivers to it.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This patch adds DEV_TO_DEV support for i.MX SDMA driver to support data
transfer between two peripheral FIFOs.
The per_2_per script requires two peripheral addresses and two DMA
requests, and it need to check the src addr and dst addr is in the SPBA
bus space or in the AIPS bus space.
Signed-off-by: NShengjiu Wang <shengjiu.wang@freescale.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Use irq_desc_get_xxx() to avoid redundant lookup of irq_desc while we
already have a pointer to corresponding irq_desc.

This is also a preparation for the removal of the 'irq' argument from
interrupt flow handlers.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: dmaengine@vger.kernel.org
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Chained irq handlers usually set up handler data as well. We now have
a function to set both under irq_desc->lock. Replace the two calls
with one.

Search and conversion was done with coccinelle.
Reported-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Cc: Julia Lawall <Julia.Lawall@lip6.fr>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: dmaengine@vger.kernel.org
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

All hardware accesses are done under virtual channel lock. That's why specific
channel lock is excessive and can be removed safely. This has been tested on
Intel Medfield and Merrifield.
Signed-off-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The dmaengine core assumes that async DMA devices will only be removed
when they not used anymore, or it assumes dma_async_device_unregister()
will only be called by dma driver exit routines. But this assumption is
not true for the IOAT driver, which calls dma_async_device_unregister()
from ioat_remove(). So current IOAT driver doesn't support device
hot-removal because it may cause system crash to hot-remove an inuse
IOAT device.

To support CPU socket hot-removal, all PCI devices, including IOAT
devices embedded in the socket, will be hot-removed. The idea solution
is to enhance the dmaengine core and IOAT driver to support hot-removal,
but that's too hard.

This patch implements a hack to disable IOAT devices under hotplug-capable
CPU socket so it won't break socket hot-removal.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This allows claiming of non-RAID channels as a private channel. This
prevents breakage of MDRAID using the IOATDMA channels via
async_tx but also allows agents such as NTB to claim channels
exclusively for its usages.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Removing static analysis error:-
Possible null pointer dereference: xt

Because currently xt is dereferenced before NULL check,
Thus Use it after NULL Check.
Signed-off-by: NManinder Singh <maninder1.s@samsung.com>
Reviewed-by: NVaneet Narang <v.narang@samsung.com>
Acked-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

clk_prepare_enable() may fail, so we should better check its return value
and propagate it in the case of error.

While at it, change the label 'err' to a more descriptive naming.
Signed-off-by: NFabio Estevam <fabio.estevam@freescale.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Commit 3b62286d ("dmaengine: Remove FSF mailing addresses") left Free
Software Foundation mailing address still in two files. Remove it now.
Signed-off-by: NJarkko Nikula <jarkko.nikula@linux.intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

When split BAR is enabled, the driver needs to dump out the split BAR
registers rather than the original 64bit BAR registers.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

The unsafe doorbell and scratchpad access should display reason when
WARN is called.  Otherwise we get a stack dump without any explanation.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

Printouts driver name and version to indicate what is being loaded.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

Benchmarking showed a significant performance increase with the MTU size
to 64k instead of 16k.  Change the driver default to 64k.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

Instead of using the platform code names, use the correct platform names
to identify the respective Intel NTB hardware.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

Disable DMA usage by default, since the CPU provides much better
performance with write combining.  Provide a module parameter to enable
DMA usage when offloading the memcpy is preferred.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

Changing the memory window BAR mappings to write combining significantly
boosts the performance.  We will also use memcpy that uses non-temporal
store, which showed performance improvement when doing non-cached
memcpys.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

Allocate memory for the NUMA node of the NTB device.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

Allocate memory and request the DMA channel for the same NUMA node as
the NTB device.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

When the ntb transport is connecting and waiting for the peer, the debug
console receives lots of debug level messages about the remote qp link
status being down.  Rate limit those messages.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

This is a simple debugging driver that enables the doorbell and
scratch pad registers to be read and written from the debugfs.  This
tool enables more complicated debugging to be scripted from user space.
This driver may be used to test that your ntb hardware and drivers are
functioning at a basic level.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

This is a simple ping pong driver that exercises the scratch pads and
doorbells of the ntb hardware.  This driver may be used to test that
your ntb hardware and drivers are functioning at a basic level.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

Add module parameters for the addresses to be used in B2B topology.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

Reset the link stats when the link goes down.  In particular, the TX and
RX index and count must be reset, or else the TX side will be sending
packets to the RX side where the RX side is not expecting them.  Reset
all the stats, to be consistent.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

On link down, don't advance RX index to the next entry.  The next entry
should never be valid after receiving the link down flag.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>

The same message "qp %d: Link Down\n" was printed at two locations in
ntb_transport.  Change the messages so they are distinct.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Signed-off-by: NJon Mason <jdmason@kudzu.us>