This allows access to pages allocated through the driver with optional
IOMMU mapping.

v2: Fix number of bytes copied and add write method
v3: drop check for kmap return
Original-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NTom St Denis <tom.stdenis@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The sdma wptr polling memory is not fast enough, then the sdma
wptr register will be random, and not equal to sdma rptr, which
will cause sdma engine hang when load driver, so clean up the sdma
wptr directly to fix this issue.

v2:add comment above the code and correct coding style
Reviewed-by: NXiangliang Yu <Xiangliang.Yu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NEmily Deng <Emily.Deng@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

to catch error that may schedule in atomic context early on
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Disable the workaround on imported BOs as well.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexdeucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We can use ttm->bdev instead.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexdeucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Allow evicting all BOs from the GTT domain.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexdeucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

delete those cgs interfaces:
amdgpu_cgs_alloc_gpu_mem
amdgpu_cgs_free_gpu_mem
amdgpu_cgs_gmap_gpu_mem
amdgpu_cgs_gunmap_gpu_mem
amdgpu_cgs_kmap_gpu_mem
amdgpu_cgs_kunmap_gpu_mem
Reviewed-by: NAlex Deucher <alexdeucher@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Reviewed-by: NAlex Deucher <alexdeucher@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add power profiling mode dynamic switch based on the workloads.
Currently, support Cumpute, VR, Video, 3D,power saving with Cumpute
have highest prority, power saving have lowest prority.

in manual dpm mode, driver will stop auto switch, just save the client's
requests. user can set power profiling mode through sysfs.

when exit manual dpm mode, driver will response the client's requests.
switch based on the client's prority.

v2: squash in fixes from Rex
Reviewed-by: NEvan Quan <evan.quan@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The gfx/compute profiling mode switch is only for internally
test. Not a complete solution and unexpectly upstream.
so revert it.
Reviewed-by: NEvan Quan <evan.quan@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Always set the graphics values to the max for the
asic type.  E.g., some 1 RB chips are actually 1 RB chips,
others are actually harvested 2 RB chips.

Fixes: https://bugs.freedesktop.org/show_bug.cgi?id=99353Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

Get gpu info through adev directly in powerplay
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

it is required if a platform supports PCIe root complex
core voltage reduction. After receiving this notification,
SBIOS can apply default PCIe root complex power policy.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

use adev as input parameter to create powerplay instance
directly. delete cgs wrap layer for power play create.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

For consistency with other DCE generations.

HPD IRQs appear to be working fine.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The ring status can change during GPU reset, but we still need to be
able to schedule TTM buffer moves in the meantime.

Otherwise we can ran into problems because of aborted move/fill
operations during GPU resets.

v2: still check if ring is available during direct submit.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NChunming zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

When we reset the GPU we also disable/enable the SDMA, but we don't want
to change TTM idea of the VRAM size in the middle of that.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NChunming zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Instead of setting the active VRAM size directly provide a the info if
we can use the buffer functions or not.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NChunming zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Those belong to the TTM handling.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Rather than querying it every time we need it.
Also fixes a crash in VM pass through if there is no
root bridge because the cached value fetch already checks
this properly.

v2: fix includes

Fixes: https://bugs.freedesktop.org/show_bug.cgi?id=105244Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: Rex Zhu<rezhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

The read/write pointers on sdma4 devices increment
beyond the ring size and should be masked.  Tested
on my Ryzen 2400G.
Signed-off-by: NTom St Denis <tom.stdenis@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

sometimes GPU is switched to other VFs and won't swich
back soon, so the kiq reg access will not signal within
a short period, instead of busy waiting a long time(MAX_KEQ_REG_WAIT)
and returning TMO we can istead sleep 5ms and try again
later (non irq context)

And since the waiting in kiq_r/weg is busy wait, so MAX_KIQ_REG_WAIT
shouldn't set to a long time, set it to 10ms is more appropriate.

if gpu already in reset state, don't retry the KIQ reg access
otherwise it would always hang because KIQ was already die usually.

v2:
replace schedule() with msleep() for the wait

v3:
use while loop for the wait repeating
use macros for the sleep period
more description for it

v4:
drop unused variable
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com
Reviewed-by: NPixel Ding <Pixel.Ding@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

1)create a routine "handle_vram_lost" to do the vram
recovery, and put it into amdgpu_device_reset/reset_sriov,
this way no need of the extra paramter to hold the
VRAM LOST information and the related macros can be removed.

3)show vram_recover failure if time out, and set TMO equal to
lockup_timeout if vram_recover is under SRIOV runtime mode.

4)report error if any ip reset failed for SR-IOV
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Acked-by: NAndrey Grodzovsky <andrey.grodzovsky@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

found recover_vram_from_shadow sometimes get executed
in paralle with SDMA scheduler, should stop all
schedulers before doing gpu reset/recover
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Tested-by: NAndrey Grodzovsky <andrey.grodzovsky@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Some were missing the close parens around options.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

When using CPU to update page table, we need to kmap all the PDs/PTs after
they are allocated and that requires a TLB shot down on each CPU, which is
quite heavy.

Instead, we map the whole visible VRAM to a kernel address at once. Pages
can be obtained from the offset.

v2: move the mapping base from gmc to amdgpu_mman structure, and the
    implementation in amdgpu_ttm_* functions
Signed-off-by: NAmber Lin <Amber.Lin@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

otherwise there will be DMAR reading error comes out from CP since
GFX is still alive and CPC's WPTR_POLL is still enabled, which would
lead to DMAR read error.

fix:
we can hault CPG after hw_fini, but cannot halt CPC becaues KIQ
stil need to be alive to let RLCV invoke, but its WPTR_POLL could
be disabled.
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

should use bo_create_kernel instead of split to two
function that create and pin the SA bo

issue:
before this patch, there are DMAR read error in host
side when running SRIOV test, the DMAR address dropped
in the range of SA bo.

fix:
after this cleanups of SA init and fini, above DMAR
eror gone.

v2:
keep sa_bo's fini instead of suspend, to keep
reporting error
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

the original method will change the wptr value in wb.
v2:
furthur cleanup
Signed-off-by: NEmily Deng <Emily.Deng@amd.com>
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

issue:
sometime GFX/MM ib test hit timeout under SRIOV env, root cause
is that engine doesn't come back soon enough so the current
IB test considered as timed out.

fix:
for SRIOV GFX IB test wait time need to be expanded a lot during
SRIOV runtimei mode since it couldn't really begin before GFX engine
come back.

for SRIOV MM IB test it always need more time since MM scheduling
is not go together with GFX engine, it is controled by h/w MM
scheduler so no matter runtime or exclusive mode MM IB test
always need more time.

v2:
use ring type instead of idx to judge
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

SRIOV doesn't give VF cg/pg feature so the MM's idle_work
is skipped for SR-IOV

v2:
remove superfluous changes
since idle_work is not scheduled for SR-IOV so the condition
check for SR-IOV inside idle_work also can be dropped

v3:
drop the SRIOV check in amdgpu_vce/uvd_suspend
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Emit frame size should match with corresponding function,
uvd_v6_0_enc_ring_emit_vm_flush has 5 amdgpu_ring_write
Signed-off-by: NJames Zhu <James.Zhu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

256MB is too small consider PTE/PDE shadow and TTM
eviction activity
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

WB_FREE should be put after all engines's hw_fini
done, otherwise the invalid wptr/rptr_addr would still
be used by engines which trigger abnormal bugs.

This fixes couple DMAR reading error in host side for SRIOV
after guest kmd is unloaded.
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

two reasons to switch SCRATCH reg method to WB method:

1)Because when doing IB test we don't want to involve KIQ health
status affect, and since SCRATCH register access is go through
KIQ that way GFX IB test would failed due to KIQ fail.

2)acccessing SCRATCH register cost much more time than WB method
because SCRATCH register access runs through KIQ which at least could
begin after GPU world switch back to current Guest VF
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

issue:
under SR-IOV sometimes the iB test will fail on
gfx ring

fix:
with cond_exec inserted in RB the gfx engine would
skip part packets if RLCV issue PREEMPT on gfx engine
if gfx engine is prior to COND_EXEC packet, this is
okay for regular command from UMD, but for the ib test
since the whole dma format doesn't support PREEMPT
so must remove the COND_EXEC from it.
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

since hotplug_work is initialized under the case of
no dc support
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

fix:
should do right shift on wb before clearing

cleanups:
1,should memset all wb buffer
2,set max wb number to 128 (total 4KB) is big enough
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>