Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

VMAs are about to not take references on the VMM they belong to, which
means more care is required when handling delayed unmapping.

Queuing it on the client workqueue ensures all pending VMA unmaps will
have completed before the VMM is destroyed.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

This is already handled in the top-level gem_new() ioctl in another manner,
but this will be removed in a future commit.

Ideally we'd not need to check up-front at all, and let the VMM code handle
error checking, but there are paths in the current BO management code where
this isn't possible due to map() not always being called during BO creation,
and map() calls not being allowed to fail during buffer migration.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

If the VMA is being deleted, we don't need to explicity unmap first
anymore.  The MMU code will automatically merge the operations into
a single page tree walk.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Could be useful for if/when a future GPU removes support for the GF100
PT layout.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

These are the new priviledged interfaces to the VMM backends, and expose
some functionality that wasn't previously available.

It's now possible to allocate a chunk of address-space (even all of it),
without causing page tables to be allocated up-front, and then map into
it at arbitrary locations.  This is the basic primitive used to support
features such as sparse mapping, or to allow userspace control over its
own address-space, or HMM (where the GPU driver isn't in control of the
address-space layout).

Rather than being tied to a subtle combination of memory object and VMA
properties, arguments that control map flags (ro, kind, etc) are passed
explicitly at map time.

The compatibility hacks to implement the old frontend on top of the new
driver backends have been replaced with something similar to implement
the old frontend's interfaces on top of the new frontend.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Adds support for:
- 64KiB/2MiB big page sizes (128KiB not supported by HW with new PT layout).
- System-memory PTs.
- LPTE "invalid" state.
- (Tegra) Use of video memory aperture.
- Sparse PDEs/PTEs.
- Additional blocklinear kinds.
- 49-bit address-space.

GP100 supports an entirely new 5-level page table layout that provides
an expanded 49-bit address-space.  It also supports the layout present
on previous generations, which we've been making do with until now.

This commit implements support for the new layout, and enables it by
default.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Adds support for:
- 64KiB big page size.
- System-memory PTs.
- LPTE "invalid" state.
- (Tegra) Use of video memory aperture.
- Sparse PDEs/PTEs.
- Additional blocklinear kinds.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Adds support for:
- 64KiB big page size.
- System-memory PTs.
- LPTE "invalid" state.
- (Tegra) Use of video memory aperture.

Adds support for marking LPTEs invalid, resulting in the corresponding
SPTEs being ignored, which is supposed to speed up TLB invalidates.

On The Tegra side, this will switch to using the video memory aperture
for all mappings.  The HW will still target non-coherent system memory,
but this aperture needs to be selected in order to support compression.

Tegra's instmem backend somewhat cheated to get this effect previously.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>

Adds support for:
- 64KiB big page size.
- System-memory PTs.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>