Create a sysfs bin attribute called "allocate" under the existing
"p2pmem" group. The only allowable operation on this file is the mmap()
call.

When mmap() is called on this attribute, the kernel allocates a chunk of
memory from the genalloc and inserts the pages into the VMA. The
dev_pagemap .page_free callback will indicate when these pages are no
longer used and they will be put back into the genalloc.

On device unbind, remove the sysfs file before the memremap_pages are
cleaned up. This ensures unmap_mapping_range() is called on the files
inode and no new mappings can be created.
Signed-off-by: NLogan Gunthorpe <logang@deltatee.com>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-by: NChaitanya Kulkarni <kch@nvidia.com>
Link: https://lore.kernel.org/r/20221021174116.7200-9-logang@deltatee.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use for_each_pci_dev() instead of open-coding it.  No functional change.

Link: https://lore.kernel.org/r/20220916140329.679633-1-yangyingliang@huawei.comSigned-off-by: NYang Yingliang <yangyingliang@huawei.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>

This interface is superseded by support in dma_map_sg() which now supports
heterogeneous scatterlists. There are no longer any users, so remove it.
Signed-off-by: NLogan Gunthorpe <logang@deltatee.com>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NJason Gunthorpe <jgg@nvidia.com>
Reviewed-by: NMax Gurtovoy <mgurtovoy@nvidia.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Add pci_p2pdma_map_segment() as a helper for dma_map_sg()
implementations. It takes an scatterlist segment that must point to a
pci_p2pdma struct page and will map it if the mapping requires a bus
address.

The return value indicates whether the mapping required a bus address
or whether the caller still needs to map the segment normally. If the
segment should not be mapped, -EREMOTEIO is returned.

This helper uses a state structure to track the changes to the
pgmap across calls and avoid needing to lookup into the xarray for
every page.

The prototype for the helper is added to dma-map-ops.h as it is only
useful to dma map implementations and don't need to pollute the public
pci-p2pdma header.
Signed-off-by: NLogan Gunthorpe <logang@deltatee.com>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Attempt to find the mapping type for P2PDMA pages on the first
DMA map attempt if it has not been done ahead of time.

Previously, the mapping type was expected to be calculated ahead of
time, but if pages are to come from userspace then there's no
way to ensure the path was checked ahead of time.

This change will calculate the mapping type if it hasn't pre-calculated
so it is no longer invalid to call pci_p2pdma_map_sg() before the mapping
type is calculated, so drop the WARN_ON when that is the case.
Signed-off-by: NLogan Gunthorpe <logang@deltatee.com>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

In 7b94b53d ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to
the whitelist"), Andrew Maier added Skylake-E 2031, 2032, and 2033 Root
Ports to the pci_p2pdma_whitelist[], so we assume P2PDMA between devices
below these ports works.

Previously we only checked the whitelist for a device at devfn 00.0 on the
root bus, which is often a "host bridge".  But these Skylake Root Ports may
be at any devfn and there may be no "host bridge" device.

Generalize pci_host_bridge_dev() so we check the first device on the root
bus, whether it is devfn 00.0 or a PCIe Root Port, against the whitelist.

[bhelgaas: commit log, comment]
Link: https://lore.kernel.org/r/20220410105213.690-2-shlomop@pliops.comTested-by: NMaor Gottlieb <maorg@nvidia.com>
Signed-off-by: NShlomo Pongratz <shlomop@pliops.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Andrew Maier <andrew.maier@eideticom.com>

In order to do P2P communication the bridge ID of the platform must be in
the P2P device table.

Update the P2P device table with a device ID for the 3rd Gen Intel Xeon
Scalable Processors.

Link: https://lore.kernel.org/r/20220209162801.7647-1-michael.j.ruhl@intel.comSigned-off-by: NMichael J. Ruhl <michael.j.ruhl@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NDan Williams <dan.j.williams@intel.com>

Since pci_alloc_p2pmem() has already called rcu_read_lock(), we're in an
RCU read-side critical section and don't need to take the lock again.  Use
percpu_ref_tryget_live_rcu() instead of percpu_ref_tryget_live() to save a
few cycles.

[bhelgaas: commit log]
Link: https://lore.kernel.org/r/ab80164f4d5b32f9e6240aa4863c3a147ff9c89f.1635974126.git.christophe.jaillet@wanadoo.frSigned-off-by: NChristophe JAILLET <christophe.jaillet@wanadoo.fr>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKrzysztof Wilczyński <kw@linux.com>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>

No driver is left using the external pgmap refcount, so remove the
code to support it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Link: https://lore.kernel.org/r/20211028151017.50234-1-hch@lst.deSigned-off-by: NDan Williams <dan.j.williams@intel.com>

strtobool() is a wrapper around kstrtobool() that has been added for
backward compatibility.

There is no reason to use the old API, so use kstrtobool() directly.

Related: ef951599 ("lib: move strtobool() to kstrtobool()")

Link: https://lore.kernel.org/r/20210915230127.2495723-3-kw@linux.comSigned-off-by: NKrzysztof Wilczyński <kw@linux.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The bus offset is bus address - physical address, so the calculation in
__pci_p2pdma_map_sg should be: bus address = physical address + bus offset.

Correct the dma_address computation in __pci_p2pdma_map_sg().

Link: https://lore.kernel.org/r/20210909032528.24517-1-wanglu@dapustor.comSigned-off-by: NWang Lu <wanglu@dapustor.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>

While looking at pci_alloc_p2pmem() I found RCU protection was not properly
applied there, as pdev->p2pdma was potentially read multiple times.

Fix pci_alloc_p2pmem(), add __rcu qualifier to p2pdma field of struct
pci_dev, and fix all other accesses to this field with proper RCU verbs.

Link: https://lore.kernel.org/r/20210701095621.3129283-1-eric.dumazet@gmail.com
Fixes: 1570175a ("PCI/P2PDMA: track pgmap references per resource, not globally")
Signed-off-by: NEric Dumazet <edumazet@google.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Ira Weiny <ira.weiny@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: "Jérôme Glisse" <jglisse@redhat.com>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>

Merge __calc_map_type_and_dist() and calc_map_type_and_dist_warn() into
calc_map_type_and_dist() to simplify the code a bit.  This now means we add
the devfn strings to the acs_buf unconditionally even if the buffer is not
printed, but that is not a lot of overhead and keeps the code much simpler.

Link: https://lore.kernel.org/r/20210614055310.3960791-1-hch@lst.deSigned-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>

In order to use upstream_bridge_distance_warn() from a dma_map function, it
must not sleep. However, pci_get_slot() takes the pci_bus_sem so it might
sleep.

In order to avoid this, try to get the host bridge's device from the first
element in the device list. It should be impossible for the host bridge's
device to go away while references are held on child devices, so the first
element should not be able to change and, thus, this should be safe.

Introduce a static function called pci_host_bridge_dev() to obtain the host
bridge's root device.

Link: https://lore.kernel.org/r/20210610160609.28447-7-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

All callers of pci_p2pdma_map_type() have a struct dev_pgmap and a struct
device (of the client doing the DMA transfer). Thus move the conversion to
struct pci_devs for the provider and client into this function.

Link: https://lore.kernel.org/r/20210610160609.28447-6-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

If the host bridge is not in the whitelist print a warning in the
calc_map_type_and_dist_warn() path detailing the vendor and device IDs that
would need to be added to the whitelist.
Suggested-by: NDon Dutile <ddutile@redhat.com>
Link: https://lore.kernel.org/r/20210610160609.28447-5-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Instead of using an int for the return value of this function, use the
correct enum pci_p2pdma_map_type.

Link: https://lore.kernel.org/r/20210610160609.28447-4-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

In order to call the calc_map_type_and_dist_warn() function from a dma_map
operation, the function must not sleep. The only reason it sleeps is to
allocate memory for the seq_buf to print a verbose warning telling the user
how to disable ACS for that path.

Instead of allocating the memory with kmalloc(), allocate a smaller buffer
on the stack. A 128 byte buffer is enough to print 10 PCI device names. A
system with 10 bridge ports between two devices that have ACS enabled would
be unusually large, so this should still be a reasonable limit.

This also cleans up the awkward (and broken) return with -ENOMEM which
contradicts the return type and the caller was not prepared for.

Link: https://lore.kernel.org/r/20210610160609.28447-3-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The function upstream_bridge_distance() has evolved such that its name is
no longer entirely reflective of what the function does.  It not only
calculates the distance between two peers but also calculates how the DMA
addresses for those two peers should be mapped.

Rename it to calc_map_type_and_dist() and rework the documentation to
better describe the two pieces of information the function returns.

[bhelgaas: tweak comment wording]
Link: https://lore.kernel.org/r/20210610160609.28447-2-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The sysfs_emit() and sysfs_emit_at() functions were introduced to make
it less ambiguous which function is preferred when writing to the output
buffer in a device attribute's "show" callback [1].

Convert the PCI sysfs object "show" functions from sprintf(), snprintf()
and scnprintf() to sysfs_emit() and sysfs_emit_at() accordingly, as the
latter is aware of the PAGE_SIZE buffer and correctly returns the number
of bytes written into the buffer.

No functional change intended.

[1] Documentation/filesystems/sysfs.rst

Related commit: ad025f8e ("PCI/sysfs: Use sysfs_emit() and
sysfs_emit_at() in "show" functions").

Link: https://lore.kernel.org/r/20210603000112.703037-2-kw@linux.comSigned-off-by: NKrzysztof Wilczyński <kw@linux.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>

Update kernel-doc so the names in the doc match the prototypes.

Link: https://lore.kernel.org/r/f19caf7a68f8365c8b573a42b4ac89ec21925c73.1603469755.git.mchehab+huawei@kernel.orgSigned-off-by: NMauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Remove the pointless paddr variable that was only used once.

Link: https://lore.kernel.org/r/20201106181941.1878556-10-hch@lst.deSigned-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Signed-off-by: NJason Gunthorpe <jgg@nvidia.com>

Now that all users of dma_virt_ops are gone we can remove the workaround
for it in the PCI peer to peer code.

Link: https://lore.kernel.org/r/20201106181941.1878556-9-hch@lst.deSigned-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Signed-off-by: NJason Gunthorpe <jgg@nvidia.com>

In support of device-dax growing the ability to front physically
dis-contiguous ranges of memory, update devm_memremap_pages() to track
multiple ranges with a single reference counter and devm instance.

Convert all [devm_]memremap_pages() users to specify the number of ranges
they are mapping in their 'struct dev_pagemap' instance.
Signed-off-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Vishal Verma <vishal.l.verma@intel.com>
Cc: Vivek Goyal <vgoyal@redhat.com>
Cc: Dave Jiang <dave.jiang@intel.com>
Cc: Ben Skeggs <bskeggs@redhat.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Ira Weiny <ira.weiny@intel.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
Cc: "Jérôme Glisse" <jglisse@redhat.co
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brice Goglin <Brice.Goglin@inria.fr>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Hulk Robot <hulkci@huawei.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Jason Yan <yanaijie@huawei.com>
Cc: Jeff Moyer <jmoyer@redhat.com>
Cc: "Jérôme Glisse" <jglisse@redhat.com>
Cc: Jia He <justin.he@arm.com>
Cc: Joao Martins <joao.m.martins@oracle.com>
Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Cc: kernel test robot <lkp@intel.com>
Cc: Mike Rapoport <rppt@linux.ibm.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Cc: Wei Yang <richard.weiyang@linux.alibaba.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lkml.kernel.org/r/159643103789.4062302.18426128170217903785.stgit@dwillia2-desk3.amr.corp.intel.com
Link: https://lkml.kernel.org/r/160106116293.30709.13350662794915396198.stgit@dwillia2-desk3.amr.corp.intel.comSigned-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The 'struct resource' in 'struct dev_pagemap' is only used for holding
resource span information.  The other fields, 'name', 'flags', 'desc',
'parent', 'sibling', and 'child' are all unused wasted space.

This is in preparation for introducing a multi-range extension of
devm_memremap_pages().

The bulk of this change is unwinding all the places internal to libnvdimm
that used 'struct resource' unnecessarily, and replacing instances of
'struct dev_pagemap'.res with 'struct dev_pagemap'.range.

P2PDMA had a minor usage of the resource flags field, but only to report
failures with "%pR".  That is replaced with an open coded print of the
range.

[dan.carpenter@oracle.com: mm/hmm/test: use after free in dmirror_allocate_chunk()]
  Link: https://lkml.kernel.org/r/20200926121402.GA7467@kadamSigned-off-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>	[xen]
Cc: Paul Mackerras <paulus@ozlabs.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Vishal Verma <vishal.l.verma@intel.com>
Cc: Vivek Goyal <vgoyal@redhat.com>
Cc: Dave Jiang <dave.jiang@intel.com>
Cc: Ben Skeggs <bskeggs@redhat.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Ira Weiny <ira.weiny@intel.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
Cc: "Jérôme Glisse" <jglisse@redhat.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brice Goglin <Brice.Goglin@inria.fr>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Hulk Robot <hulkci@huawei.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Jason Yan <yanaijie@huawei.com>
Cc: Jeff Moyer <jmoyer@redhat.com>
Cc: Jia He <justin.he@arm.com>
Cc: Joao Martins <joao.m.martins@oracle.com>
Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Cc: kernel test robot <lkp@intel.com>
Cc: Mike Rapoport <rppt@linux.ibm.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Cc: Wei Yang <richard.weiyang@linux.alibaba.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lkml.kernel.org/r/159643103173.4062302.768998885691711532.stgit@dwillia2-desk3.amr.corp.intel.com
Link: https://lkml.kernel.org/r/160106115761.30709.13539840236873663620.stgit@dwillia2-desk3.amr.corp.intel.comSigned-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

sg_init_table() zeroes its first argument, so the allocation of that
argument doesn't have to.

The semantic patch that makes this change is as follows:
(http://coccinelle.lip6.fr/)

  // <smpl>
  @@
  expression x;
  @@

  x =
  - kzalloc
  + kmalloc
   (...)
  ...
  sg_init_table(x,...)
  // </smpl>

Link: https://lore.kernel.org/r/1600601186-7420-15-git-send-email-Julia.Lawall@inria.frSigned-off-by: NJulia Lawall <Julia.Lawall@inria.fr>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Sysfs "show" methods should return the number of bytes printed into the
buffer.  This is the return value of scnprintf() [1].

snprintf(buf, size, ...) prints at most "size" bytes into "buf", but
returns the number of bytes that *would* be printed if "buf" were large
enough.

Replace use of snprintf() with scnprintf().  No functional change intended.

Related:
  https://patchwork.kernel.org/patch/9946759/#20969333
  https://lwn.net/Articles/69419

[1] Documentation/filesystems/sysfs.rst

[bhelgaas: squashed, commit log]
Suggested-by: NBjorn Helgaas <bhelgaas@google.com>
Link: https://lore.kernel.org/r/20200824233918.26306-2-kw@linux.com
Link: https://lore.kernel.org/r/20200824233918.26306-3-kw@linux.com
Link: https://lore.kernel.org/r/20200824233918.26306-4-kw@linux.comSigned-off-by: NKrzysztof Wilczyński <kw@linux.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

My commit to make DMA ops support optional missed the reference in
the p2pdma code.  And while the build bot didn't manage to find a config
where this can happen, Matthew did.  Fix this by replacing two IS_ENABLED
checks with ifdefs.

Fixes: 2f9237d4 ("dma-mapping: make support for dma ops optional")
Link: https://lore.kernel.org/r/20200810124843.1532738-1-hch@lst.deReported-by: NMatthew Wilcox <willy@infradead.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>

Allow P2PDMA if the CPU vendor is AMD and family is 0x17 (Zen) or greater.

[bhelgaas: commit log, simplify #if/#else/#endif]
Link: https://lore.kernel.org/r/20200729231844.4653-1-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>

Currently the ACS capability is being looked up at a number of places. Read
and store it once at enumeration so that it can be used by all later.  No
functional change intended.

Link: https://lore.kernel.org/r/20200707224604.3737893-2-rajatja@google.comSigned-off-by: NRajat Jain <rajatja@google.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

According to the hardware architect, pre-Zen parts support p2p writes and
Zen parts support both p2p reads and writes.

Add entries for Zen parts Raven (0x15d0) and Renoir (0x1630).

Link: https://lore.kernel.org/r/20200406194201.846411-1-alexander.deucher@amd.comSigned-off-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Acked-by: NHuang Rui <ray.huang@amd.com>

Add the three remaining Intel Sky Lake-E host Root Ports to the whitelist
of p2pdma.

P2P has been tested and is working on this system.

Link: https://lore.kernel.org/r/20200207221219.4309-1-andrew.maier@eideticom.comSigned-off-by: NAndrew Maier <andrew.maier@eideticom.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>

Intel SkyLake-E was successfully tested for p2pdma transactions spanning
over a host bridge and PCI bridge with IOMMU on.  Add it to the whitelist.

Link: https://lore.kernel.org/r/1575669165-31697-1-git-send-email-abaloyan@gigaio.comSigned-off-by: NArmen Baloyan <abaloyan@gigaio.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>

The comment describing pci_p2pdma_distance_many() still referred to
the devices being behind the same root port. This no longer applies
so reword the documentation.

Link: https://lore.kernel.org/r/20190730163545.4915-15-logang@deltatee.com
Link: https://lore.kernel.org/r/20190812173048.9186-15-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

Now that we map the requests correctly we can remove the iommu_present()
restriction.

Link: https://lore.kernel.org/r/20190730163545.4915-14-logang@deltatee.com
Link: https://lore.kernel.org/r/20190812173048.9186-14-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

Any requests that traverse the host bridge will need to be mapped into the
IOMMU, so call dma_map_sg() inside pci_p2pdma_map_sg() when appropriate.

Similarly, call dma_unmap_sg() inside pci_p2pdma_unmap_sg().

Link: https://lore.kernel.org/r/20190730163545.4915-13-logang@deltatee.com
Link: https://lore.kernel.org/r/20190812173048.9186-13-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

When upstream_bridge_distance() is called, store the method required to map
the DMA transfers in an xarray so it can be looked up efficiently on the
hot path in pci_p2pdma_map_sg().

Link: https://lore.kernel.org/r/20190730163545.4915-12-logang@deltatee.com
Link: https://lore.kernel.org/r/20190812173048.9186-12-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

Factor out the bus-only mapping into its own static function.  No
functional changes. The original pci_p2pdma_map_sg_attrs() will be used to
decide whether this is an appropriate way to map.

Link: https://lore.kernel.org/r/20190730163545.4915-11-logang@deltatee.com
Link: https://lore.kernel.org/r/20190812173048.9186-11-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

Add pci_p2pdma_unmap_sg() to the two places that call pci_p2pdma_map_sg().

This is a prep patch to introduce correct mappings for p2pdma transactions
that go through the root complex.

Link: https://lore.kernel.org/r/20190730163545.4915-10-logang@deltatee.com
Link: https://lore.kernel.org/r/20190812173048.9186-10-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

This is to match the dma_map_sg() API which this function will have to call
in an future patch.

Add a pci_p2pdma_map_sg_attrs() function and helper to call it with no
attributes just like the dma_map_sg() function.

Link: https://lore.kernel.org/r/20190730163545.4915-9-logang@deltatee.com
Link: https://lore.kernel.org/r/20190812173048.9186-9-logang@deltatee.comSigned-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>