A few architectures support uncached kernel segments.  In that case we get
an uncached mapping for a given physica address by using an offset in the
uncached segement.  Implement support for this scheme in the generic
dma-direct code instead of duplicating it in arch hooks.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Virtual addresses return from dma(m)_alloc_coherent are opaque in what
backs then, and drivers must not poke into them.  Switch the driver
to use the generic DMA API mmap helper to avoid these games.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NBartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>

This export is not used in modular code, which is a good thing as
everyone should use the proper DMA API instead.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NPaul Burton <paul.burton@mips.com>

With the addition of the local memory allocator, the HCD_LOCAL_MEM
flag can be dropped and the checks against it replaced with a check
for the localmem_pool ptr being initialized.
Signed-off-by: NLaurentiu Tudor <laurentiu.tudor@nxp.com>
Tested-by: NFredrik Noring <noring@nocrew.org>
Reviewed-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

In preparation for dropping the existing "coherent" dma mem declaration
APIs, replace the current dma_declare_coherent_memory() based mechanism
with the creation of a genalloc pool that will be used in the OHCI
subsystem as replacement for the DMA APIs.
Signed-off-by: NLaurentiu Tudor <laurentiu.tudor@nxp.com>
Reviewed-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

In preparation for dropping the existing "coherent" dma mem declaration
APIs, replace the current dma_declare_coherent_memory() based mechanism
with the creation of a genalloc pool that will be used in the OHCI
subsystem as replacement for the DMA APIs.
Signed-off-by: NLaurentiu Tudor <laurentiu.tudor@nxp.com>
Reviewed-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

For HCs that have local memory, replace the current DMA API usage with
a genalloc generic allocator to manage the mappings for these devices.
To help users, introduce a new HCD API, usb_hcd_setup_local_mem() that
will setup up the genalloc backing up the device local memory. It will
be used in subsequent patches.  This is in preparation for dropping
the existing "coherent" dma mem declaration APIs.  The current
implementation was relying on a short circuit in the DMA API that in
the end, was acting as an allocator for these type of devices.
Signed-off-by: NLaurentiu Tudor <laurentiu.tudor@nxp.com>
Tested-by: NFredrik Noring <noring@nocrew.org>
Reviewed-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

gen_pool_dma_zalloc() is a zeroed memory variant of
gen_pool_dma_alloc().  Also document the return values of both, and
indicate NULL as a "%NULL" constant.
Signed-off-by: NFredrik Noring <noring@nocrew.org>
Reviewed-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Commit fdaeec198ada ("dma-contiguous: add dma_{alloc,free}_contiguous()
helpers") adds a pair of new helper functions so as to abstract code in
the dma-direct (and other places in the future), however it breaks QEMU
boot feature using x86_64 defconfig.

That's because x86_64 defconfig has CONFIG_DMA_CMA=n so those two newly
introduced helper functions are empty in their !CONFIG_DMA_CMA version,
while previously the platform independent dma-direct code had fallback
alloc_pages_node() and __free_pages().

So this patch fixes it by adding alloc_pages_node() and __free_pages()
in the !CONFIG_DMA_CMA version of the two helper functions.

Tested with below QEMU command:
  qemu-system-x86_64 -m 512m \
      -drive file=images/x86_64/rootfs.ext4,format=raw,if=ide \
      -append 'console=ttyS0 root=/dev/sda' -nographic \
      -kernel arch/x86_64/boot/bzImage

with the rootfs from the below link:
  https://github.com/ClangBuiltLinux/continuous-integration/raw/master/images/x86_64/rootfs.ext4

Fixes: fdaeec198ada ("dma-contiguous: add dma_{alloc,free}_contiguous() helpers")
Reported-by: NNathan Chancellor <natechancellor@gmail.com>
Signed-off-by: NNicolin Chen <nicoleotsuka@gmail.com>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

The addresses within a single page are always contiguous, so it's
not so necessary to always allocate one single page from CMA area.
Since the CMA area has a limited predefined size of space, it may
run out of space in heavy use cases, where there might be quite a
lot CMA pages being allocated for single pages.

However, there is also a concern that a device might care where a
page comes from -- it might expect the page from CMA area and act
differently if the page doesn't.

This patch tries to use the fallback alloc_pages path, instead of
one-page size allocations from the global CMA area in case that a
device does not have its own CMA area. This'd save resources from
the CMA global area for more CMA allocations, and also reduce CMA
fragmentations resulted from trivial allocations.
Signed-off-by: NNicolin Chen <nicoleotsuka@gmail.com>
Tested-by: Ndann frazier <dann.frazier@canonical.com>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Both dma_alloc_from_contiguous() and dma_release_from_contiguous() are
very simply implemented, but requiring callers to pass certain
parameters like count and align, and taking a boolean parameter to check
__GFP_NOWARN in the allocation flags. So every function call duplicates
similar work:

	unsigned long order = get_order(size);
	size_t count = size >> PAGE_SHIFT;

	page = dma_alloc_from_contiguous(dev, count, order,
			gfp & __GFP_NOWARN);

	[...]

	dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT);

Additionally, as CMA can be used only in the context which permits
sleeping, most of callers do a gfpflags_allow_blocking() check and a
corresponding fallback allocation of normal pages upon any false result:

	if (gfpflags_allow_blocking(flag))
		page = dma_alloc_from_contiguous();
	if (!page)
		page = alloc_pages();

	[...]

	if (!dma_release_from_contiguous(dev, page, count))
		__free_pages(page, get_order(size));

So this patch simplifies those function calls by abstracting these
operations into the two new functions: dma_{alloc,free}_contiguous.

As some callers of dma_{alloc,release}_from_contiguous() might be
complicated, this patch just implements these two new functions to
kernel/dma/direct.c only as an initial step.
Suggested-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NNicolin Chen <nicoleotsuka@gmail.com>
Tested-by: Ndann frazier <dann.frazier@canonical.com>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Clang warns:

drivers/iommu/dma-iommu.c:897:6: warning: logical not is only applied to
the left hand side of this comparison [-Wlogical-not-parentheses]
        if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0)
            ^                                 ~~
drivers/iommu/dma-iommu.c:897:6: note: add parentheses after the '!' to
evaluate the comparison first
        if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0)
            ^
             (                                    )
drivers/iommu/dma-iommu.c:897:6: note: add parentheses around left hand
side expression to silence this warning
        if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0)
            ^
            (                                )
1 warning generated.

Judging from the rest of the commit and the conditional in
iommu_dma_map_sg, either

    if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC))

or
    if ((attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0)

was intended, not a combination of the two.

I personally think that the former is easier to understand so use that.

Fixes: 06d60728 ("iommu/dma: move the arm64 wrappers to common code")
Link: https://github.com/ClangBuiltLinux/linux/issues/497Signed-off-by: NNathan Chancellor <natechancellor@gmail.com>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

With most of the previous functionality now elsewhere a lot of the
headers included in this file are not needed.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NRobin Murphy <robin.murphy@arm.com>
Reviewed-by: NMukesh Ojha <mojha@codeaurora.org>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

For entirely dma coherent architectures there is no requirement to ever
remap dma coherent allocation.  Move all the remap and pool code under
IS_ENABLED() checks and drop the Kconfig dependency.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Inline __iommu_dma_mmap_pfn into the main function, and use the
fact that __iommu_dma_get_pages return NULL for remapped contigous
allocations to simplify the code flow a bit.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Inline __iommu_dma_get_sgtable_page into the main function, and use the
fact that __iommu_dma_get_pages return NULL for remapped contigous
allocations to simplify the code flow a bit.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

All the logic in iommu_dma_alloc that deals with page allocation from
the CMA or page allocators can be split into a self-contained helper,
and we can than map the result of that or the atomic pool allocation
with the iommu later.  This also allows reusing __iommu_dma_free to
tear down the allocations and MMU mappings when the IOMMU mapping
fails.

Based on a patch from Robin Murphy.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Most importantly clear up the size / iosize confusion.  Also rename addr
to cpu_addr to match the surrounding code and make the intention a little
more clear.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
[hch: split from a larger patch]
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Most of it can double up to serve the failure cleanup path for
iommu_dma_alloc().
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Instead of having a separate code path for the non-blocking alloc_pages
and CMA allocations paths merge them into one.  There is a slight
behavior change here in that we try the page allocator if CMA fails.
This matches what dma-direct and other iommu drivers do and will be
needed to use the dma-iommu code on architectures without DMA remapping
later on.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Always remapping CMA allocations was largely a bodge to keep the freeing
logic manageable when it was split between here and an arch wrapper. Now
that it's all together and streamlined, we can relax that limitation.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Shuffle around the self-contained atomic and non-contiguous cases to
return early and get out of the way of the CMA case that we're about to
work on next.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
[hch: slight changes to the code flow]
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The freeing logic was made particularly horrible by part of it being
opaque to the arch wrapper, which led to a lot of convoluted repetition
to ensure each path did everything in the right order. Now that it's
all private, we can pick apart and consolidate the logically-distinct
steps of freeing the IOMMU mapping, the underlying pages, and the CPU
remap (if necessary) into something much more manageable.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
[various cosmetic changes to the code flow]
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

We only have a single caller of this function left, so open code it there.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Move the call to dma_common_pages_remap into __iommu_dma_alloc and
rename it to iommu_dma_alloc_remap.  This creates a self-contained
helper for remapped pages allocation and mapping.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Since we duplicate the find_vm_area() logic a few times in places where
we only care aboute the pages, factor out a helper to abstract it.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
[hch: don't warn when not finding a region, as we'll rely on that later]
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The remaining internal callsites don't care about having prototypes
compatible with the relevant dma_map_ops callbacks, so the extra
level of indirection just wastes space and complictaes things.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Most of the callers don't care, and the couple that do already have the
domain to hand for other reasons are in slow paths where the (trivial)
overhead of a repeated lookup will be utterly immaterial.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
[hch: dropped the hunk touching iommu_dma_get_msi_page to avoid a
 conflict with another series]
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Moving this function up to its unmap counterpart helps to keep related
code together for the following changes.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

There is nothing really arm64 specific in the iommu_dma_ops
implementation, so move it to dma-iommu.c and keep a lot of symbols
self-contained.  Note the implementation does depend on the
DMA_DIRECT_REMAP infrastructure for now, so we'll have to make the
DMA_IOMMU support depend on it, but this will be relaxed soon.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

arch_dma_prep_coherent can handle physically contiguous ranges larger
than PAGE_SIZE just fine, which means we don't need a page-based
iterator.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

We now have a arch_dma_prep_coherent architecture hook that is used
for the generic DMA remap allocator, and we should use the same
interface for the dma-iommu code.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

No need for a __KERNEL__ guard outside uapi and add a missing comment
describing the #else cpp statement.  Last but not least include
<linux/errno.h> instead of the asm version, which is frowned upon.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Pull tracing warning fix from Steven Rostedt:
 "Make the GCC 9 warning for sub struct memset go away.

  GCC 9 now warns about calling memset() on partial structures when it
  goes across multiple fields. This adds a helper for the place in
  tracing that does this type of clearing of a structure"

* tag 'trace-v5.2-rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace:
  tracing: Silence GCC 9 array bounds warning

Pull KVM fixes from Paolo Bonzini:
 "The usual smattering of fixes and tunings that came in too late for
  the merge window, but should not wait four months before they appear
  in a release.

  I also travelled a bit more than usual in the first part of May, which
  didn't help with picking up patches and reports promptly"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (33 commits)
  KVM: x86: fix return value for reserved EFER
  tools/kvm_stat: fix fields filter for child events
  KVM: selftests: Wrap vcpu_nested_state_get/set functions with x86 guard
  kvm: selftests: aarch64: compile with warnings on
  kvm: selftests: aarch64: fix default vm mode
  kvm: selftests: aarch64: dirty_log_test: fix unaligned memslot size
  KVM: s390: fix memory slot handling for KVM_SET_USER_MEMORY_REGION
  KVM: x86/pmu: do not mask the value that is written to fixed PMUs
  KVM: x86/pmu: mask the result of rdpmc according to the width of the counters
  x86/kvm/pmu: Set AMD's virt PMU version to 1
  KVM: x86: do not spam dmesg with VMCS/VMCB dumps
  kvm: Check irqchip mode before assign irqfd
  kvm: svm/avic: fix off-by-one in checking host APIC ID
  KVM: selftests: do not blindly clobber registers in guest asm
  KVM: selftests: Remove duplicated TEST_ASSERT in hyperv_cpuid.c
  KVM: LAPIC: Expose per-vCPU timer_advance_ns to userspace
  KVM: LAPIC: Fix lapic_timer_advance_ns parameter overflow
  kvm: vmx: Fix -Wmissing-prototypes warnings
  KVM: nVMX: Fix using __this_cpu_read() in preemptible context
  kvm: fix compilation on s390
  ...

Pull /dev/random fix from Ted Ts'o:
 "Fix a soft lockup regression when reading from /dev/random in early
  boot"

* tag 'random_for_linus_stable' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/random:
  random: fix soft lockup when trying to read from an uninitialized blocking pool

Fixes: eb9d1bf0: "random: only read from /dev/random after its pool has received 128 bits"
Reported-by: Nkernel test robot <lkp@intel.com>
Signed-off-by: NTheodore Ts'o <tytso@mit.edu>