Make the code easier to understand.
Signed-off-by: NAlex Xie <AlexBin.Xie@amd.com>
Reviewed-by: NMichel Dänzer <michel.daenzer@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

  Move several if statements and a loop statment from
  run time to initialization time.
Signed-off-by: NAlex Xie <AlexBin.Xie@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This change is also useful for the upcoming changes where page tables
can be updated by CPU.
Signed-off-by: NHarish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

If updating the PDs fails we now invalidate all entries to try again later.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Rename adjust_mc_addr to get_vm_pde and check the address bits in one place.

v2: handle vcn as well, keep setting the valid bit manually,
    add a BUG_ON() for GMC v6, v7 and v8 as well.
v3: handle vcn_v1_0_enc_ring_emit_vm_flush as well.
v4: fix the BUG_ON mask for GFX6-8
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

That GFX9 needs a PDE in the registers is entirely GFX9 specific.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

partial revert commit <6971d3d> - drm/amdgpu: cleanup logic in
amdgpu_vm_flush
Signed-off-by: NFlora Cui <Flora.Cui@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <David1.Zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

before that, we have function to check if reset happens by using reset count.
v2: always update reset count after vm flush
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

v2: directly return for 'if' case.
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Implement the vmid reservation.

v2: move sync waiting only when flush needs
v3: fix racy
v4: peek fence instead of get fence, and fix potential context starved.
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Limit reserved vmids to 1 to avoid taking too many
out of commission and starving the system.

v2: move #define to amdgpu_vm.h
v3: move reserved vmid counter to id_manager,
and increase counter before allocating vmid
v4: rename to reserved_vmid_num
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

add reserve/unreserve vmid funtions. Used to reserve
vmids for certain shader debugging functionality that
required a fixed vmid for the life of the debug.

v3:
only reserve vmid from gfxhub
v4:
fix racy condition
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

v2: rename dedicated_vmid to reserved_vmid
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

It will be used for reserving vmid for shader debugging
that requires a fixed vmid.

v2: fix warning (Alex)
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

[  338.384770] BUG: unable to handle kernel NULL pointer dereference at           (null)
[  338.384817] IP: [<          (null)>]           (null)
[  338.385505] RIP: 0010:[<0000000000000000>]  [<          (null)>]           (null)
[  338.385950] Call Trace:
[  338.385993]  [<ffffffffa05d2313>] ? amdgpu_vm_flush+0x283/0x400 [amdgpu]
[  338.386025]  [<ffffffff811818d3>] ? printk+0x4d/0x4f
[  338.386074]  [<ffffffffa05d4906>] amdgpu_ib_schedule+0x4a6/0x4d0 [amdgpu]
[  338.386140]  [<ffffffffa0673e54>] amdgpu_job_run+0x64/0x180 [amdgpu]
[  338.386203]  [<ffffffffa0672e09>] amd_sched_main+0x2e9/0x4a0 [amdgpu]
[  338.386232]  [<ffffffff810bfce0>] ? prepare_to_wait_event+0x110/0x110
[  338.386295]  [<ffffffffa0672b20>] ? amd_sched_select_entity+0xe0/0xe0 [amdgpu]
[  338.386327]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[  338.386349]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[  338.386376]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[  338.386401] Code:  Bad RIP value.
[  338.386420] RIP  [<          (null)>]           (null)
[  338.386443]  RSP <ffffc90001bd7d40>
[  338.386458] CR2: 0000000000000000
[  338.398508] ---[ end trace 4c66fcdc74b9a0a2 ]---
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Reinitializing the VM manager during suspend/resume is a very very bad
idea since all the VMs are still active and kicking.

This can lead to random VM faults after resume when new processes
become the same client ID assigned.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

The problem is that executing the jobs in the right order doesn't give you the right result
because consecutive jobs executed on the same engine are pipelined.
In other words job B does it buffer read before job A has written it's result.
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Fix PRT handling on gfx9

v2: unify PRT bit for all ASICs
v3: move PRT flag checking in amdgpu_vm_bo_split_mapping()
Signed-off-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Acked-by: NDavid Zhou <david1.zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Use amdgpu_vm_bo_update_mapping() instead of amdgpu_vm_bo_split_mapping() here.

We don't want any flags set in the cleared areas and splitting
shouldn't be necessary.

v2: fix typo in commit message
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

Trace on which hub we are doing the flush.

v2: fix typo in commit message
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Trace on which VMHUB we assigned an VMID.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Enable concurrent VM flushes for Vega10.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Drop invalidating both hubs from each engine.

v2: don't use hardcoded values
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This way GFX and MM won't fight for VMIDs any more.

Initially disabled since we need to stop flushing all HUBS
at the same time as well.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

David suggested this a long time ago, instead of checking
each ring just walk over all the VMIDs in reverse LRU order.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

By default, the value is set by individual gmc.
if a specific value is input, it overrides the global value for all

v2: create helper funcs
v3: update gmc9 APU's num_level athough it may be updated in the future.
Signed-off-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Set reasonable defaults per family.

v2: set both of them in gmc
v3: move vm size and block size in vm manager
v4: squash in warning fix from Alex Xie
v5: squash in min() warning fix
Signed-off-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We should probably rename amdgpu_gart_funcs sooner or later.
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Remove some of the extra checks where they don't hurt us.
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Abort early if there is nothing todo and correctly indent the "if"s.
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The name is a bit confusing and the extra "? true : false" is superflous.

Additional to that remove setting the reset counter directly after checking it.
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Not used any more.
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We accidentally dereference "cb" if the kmalloc() fails.

Fixes: 451bc8eb ("drm/amdgpu: fix PRT teardown on VM fini v3")
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NHarry Wentland <harry.wentland@amd.com>
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This only makes a difference for 32-bit systems. The idea is to have a
fixed virtual address space size with 4-level page tables and to
minimize differences between 32 and 64-bit systems.

v2: Update commit message.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Fix the start/end address calculation for address ranges that span
multiple page directories in amdgpu_vm_alloc_levels.

Add error messages if page tables aren't found. Otherwise the page
table update would just fail silently.

v2:
 * Change WARN_ON to WARN_ON_ONCE
 * Move masking of high address bits to caller
 * Add range-check for "from" and "to"
v3:
 * Replace WARN_ON_ONCE in get_pt with pr_err in caller
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

With 4-level page tables the maximum VM size is 256TB. That's 64G
pages, which can't be represented in 32-bit.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

1) Adapt to vulkan:
Now use double SWITCH BUFFER to replace the 128 nops w/a,
because when vulkan introduced, umd can insert 7 ~ 16 IBs
per submit which makes 256 DW size cannot hold the whole
DMAframe (if we still insert those 128 nops), CP team suggests
use double SWITCH_BUFFERs, instead of tricky 128 NOPs w/a.

2) To fix the CE VM fault issue when MCBP introduced:
Need one more COND_EXEC wrapping IB part (original one us
for VM switch part).

this change can fix vm fault issue caused by below scenario
without this change:

>CE passed original COND_EXEC (no MCBP issued this moment),
 proceed as normal.

>DE catch up to this COND_EXEC, but this time MCBP issued,
 thus DE treats all following packages as NOP. The following
 VM switch packages now looks just as NOP to DE, so DE
 dosen't do VM flush at all.

>Now CE proceeds to the first IBc, and triggers VM fault,
 because DE didn't do VM flush for this DMAframe.

3) change estimated alloc size for gfx9.
with new DMAframe scheme, we need modify emit_frame_size
for gfx9

4) No need to insert 128 nops after gfx8 vm flush anymore
because there was double SWITCH_BUFFER append to vm flush,
and for gfx7 we already use double SWITCH_BUFFER following
after vm_flush so no change needed for it.

5) Change emit_frame_size for gfx8

v2: squash in BUG removal from Monk
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Allocate and free page directories on demand.

V2:
a. clear entries allocation
b. fix entries index calculation
c. need alloc sub level even parent bo was allocated

Signed-off-by: Christian König <christian.koenig@amd.com> (v1)
Reviewed-by: Alex Deucher <alexander.deucher@amd.com> (v1)
Signed-off-by: Chunming Zhou <David1.Zhou@amd.com> (v2)
Acked-by: Alex Deucher <alexander.deucher@amd.com> (v2)
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Not the best solution, but good enough for now.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Update all levels of the page directory.

V2:
a. sub level pdes always are written to incorrect place.
b. sub levels need to update regardless of parent updates.

Signed-off-by: Christian König <christian.koenig@amd.com> (V1)
Reviewed-by: Alex Deucher <alexander.deucher@amd.com> (V1)
Signed-off-by: Chunming Zhou <David1.Zhou@amd.com> (V2)
Acked-by: Alex Deucher <alexander.deucher@amd.com> (V2)
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>