Keep blocking the CS, but revert everything else. Mapping BOs and info IOCTL
are harmless and can still happen even when VRAM content ist lost.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Introduce a flag to signal that access to a BO will be synchronized
through an external mechanism.

Currently all buffers shared between contexts are subject to implicit
synchronization. However, this is only required for protocols that
currently don't support an explicit synchronization mechanism (DRI2/3).

This patch introduces the AMDGPU_GEM_CREATE_EXPLICIT_SYNC, so that
users can specify when it is safe to disable implicit sync.

v2: only disable explicit sync in amdgpu_cs_ioctl
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This didn't helped as intended, just simplify the code.

v2: unlock mmap_sem in the error path as well
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add the IOCTL interface so that applications can allocate per VM BOs.

Still WIP since not all corner cases are tested yet, but this reduces average
CS overhead for 10K BOs from 21ms down to 48us.

v2: add some extra checks, remove the WIP tag
v3: rename new flag to AMDGPU_GEM_CREATE_VM_ALWAYS_VALID
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Don't allow them to be GEM imported into another process.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Instead of validating all page tables when one was evicted,
track which one needs a validation.

v2: simplify amdgpu_vm_ready as well
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com> (v1)
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Those are certainly not kernel allocations, instead set the NO_CPU_ACCESS flag.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Stop checking the mapped BO itself, cause that one is
certainly not a page table.

Additional to that move the code into amdgpu_vm.c
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Split that into vm_bo_base and bo_va to allow other uses as well.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The shadow handling isn't implemented completely for userspace BOs and
the kernel sets the VRAM_CONTIGUOUS as necessary.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Change "prefered" to "preferred"
Signed-off-by: NKent Russell <kent.russell@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

drm_*_reference() and drm_*_unreference() functions are just
compatibility alias for drm_*_get() and drm_*_put() and should not be
used by new code. So convert all users of compatibility functions to use
the new APIs.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NCihangir Akturk <cakturk@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The parameter init_value contains the value to which we initialized
VRAM bo when AMDGPU_GEM_CREATE_VRAM_CLEARED flag is set.
Signed-off-by: NYong Zhao <Yong.Zhao@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Instead of open coding the conversion from u64 to pointers.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We can finally remove this now.

v2: remove now unused max_size variable as well.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Otherwise we trigger a bunch of WARN_ONs when this is called.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

VM is mandatory for all hw amdgpu supports. So remove the leftovers
to make it optionally.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

below ioctl will return -ENODEV:
amdgpu_cs_ioctl
amdgpu_cs_wait_ioctl
amdgpu_cs_wait_fences_ioctl
amdgpu_gem_va_ioctl
amdgpu_info_ioctl

v2: only for map and replace cases in amdgpu_gem_va_ioctl
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This kind of reset handling was removed a long time ago.

v2: fix warning (Alex)
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We need to check if the VM is swapped out before trying to update it.

Fixes: 23e0563e ("drm/amdgpu: clear freed mappings immediately when BO may be freed")
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Migration to VRAM will break the sharing, resulting in rendering on the exporting GPU never becoming
visible on the importing GPU.

v2: Don't pin BOs to GTT. Instead, refuse to migrate them out of GTT.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NChristopher James Halse Rogers <christopher.halse.rogers@canonical.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NAlex Xie <AlexBin.Xie@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Also, add the fence of the clear operations to the BO to ensure that
the underlying memory can only be re-used after all PTEs pointing to
it have been cleared.

This avoids the following sequence of events that could be triggered
by user space:

1. Submit a CS that accesses some BO _without_ adding that BO to the
   buffer list.
2. Free that BO.
3. Some other task re-uses the memory underlying the BO.
4. The CS is submitted to the hardware and accesses memory that is
   now already in use by somebody else.

By clearing the page tables immediately in step 2, a GPU VM fault will
be triggered in step 4 instead of wild memory accesses.

v2: use amdgpu_bo_fence directly
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Update all levels of the page directory.

V2:
a. sub level pdes always are written to incorrect place.
b. sub levels need to update regardless of parent updates.

Signed-off-by: Christian König <christian.koenig@amd.com> (V1)
Reviewed-by: Alex Deucher <alexander.deucher@amd.com> (V1)
Signed-off-by: Chunming Zhou <David1.Zhou@amd.com> (V2)
Acked-by: Alex Deucher <alexander.deucher@amd.com> (V2)
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NAlex Xie <AlexBin.Xie@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We will add the fence to freed buffer objects in a later commit, to ensure
that the underlying memory can only be re-used after all references in
page tables have been cleared.
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add a new operation to replace mappings in a VM with a new one.

v2: Fix Jerry's comment, separate out clear operation.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

A new VM operation to remove all mappings in a range.

v2: limit unmapped area as noted by Jerry
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This makes it easier to implement a replace operation.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This lets each asic set whichever flags it supports.
Signed-off-by: NAlex Xie <AlexBin.Xie@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Till GFX8 we can only enable PRT support globally, but with the next hardware
generation we can do this on a per page basis.

Keep the interface consistent by adding PRT mappings and enable
support globally on current hardware when the first mapping is made.

v2: disable PRT support delayed and on all error paths
v3: PRT and other permissions are mutal exclusive,
    PRT mappings don't need a BO.
v4: update PRT mappings durign CS as well, make va_flags 64bit
Signed-off-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Reject it if there are any invalid flags or domains.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This removes code duplication.
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NSamuel Pitoiset <samuel.pitoiset@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We don't add any fences do the buffer, but just use it's address.

Additional to that we don't need a duplicates list here.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Somebody could try to free the bo_va between mapping and updating it.

v2: fix typos in comment
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Skip amdgpu_gem_va_update_vm otherwise. Also clean up the check for the
non-shadow page tables using the new helper function.

This fixes a crash with the stack trace:

amdgpu_gem_va_update_vm
-> amdgpu_vm_update_page_directory
 -> amdgpu_ttm_bind
  -> amdgpu_gtt_mgr_alloc

v2: actually check bo->shadow instead of just checking bo twice
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Skip amdgpu_gem_va_update_vm when shadow the page directory is swapped out.
Clean up the check for non-shadow BOs as well using the new helper function.

This fixes a crash with the stack trace:

amdgpu_gem_va_update_vm
-> amdgpu_vm_update_page_directory
 -> amdgpu_ttm_bind
  -> amdgpu_gtt_mgr_alloc
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Instead of messing with the PD directly.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Saves a bunch of CPU cycles when swapping things back in and
allows us to split the VM headers into a separate file.

v2: rename parameters
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

It's completely pointless to have two pointers to the
device in the same structure.

v2: rename function to amdgpu_ttm_adev, fix typos
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>