Memory used by KFD applications can contain sensitive information that
should not be leaked to other processes. The current approach to prevent
leaks is to clear VRAM at allocation time. This is not effective because
memory can be reused in other ways without being cleared. Synchronously
clearing memory on the allocation path also carries a significant
performance penalty.

Stop clearing memory at allocation time. Instead mark the memory for
wipe on release.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Wipe VRAM memory containing sensitive data when moving or releasing
BOs. Clearing the memory is pipelined to minimize any impact on
subsequent memory allocation latency. Use of a poison value should
help debug future use-after-free bugs.

When moving BOs, the existing ttm_bo_pipelined_move ensures that the
memory won't be reused before being wiped.

When releasing BOs, the BO is fenced with the memory fill operation,
which results in queuing the BO for a delayed delete.

v2: Move amdgpu_amdkfd_unreserve_memory_limit into
amdgpu_bo_release_notify so that KFD can use memory that's still
being cleared in the background
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

previously the ucode loading of PSP was repreated, one executed in
phase_1 init/re-init/resume and the other in fw_loading routine

Avoid this double loading by clearing ip_blocks.status.hw in suspend or reset
prior to the FW loading and any block's hw_init/resume

v2:
still do the smu fw loading since it is needed by bare-metal

v3:
drop the change in reinit_early_sriov, just clear all block's status.hw
in the head place and set the status.hw after hw_init done is enough
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NEmily Deng <Emily.Deng@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

for SRIOV the SOS fw of PSP is loaded in hypervisor thus
guest won't tell the version of it, and judging feature by
reading the sos fw version in guest side is completely wrong
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NEmily Deng <Emily.Deng@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

we can simplify all those unnecessary function under
SRIOV for vega10 since:
1) PSP L1 policy is by force enabled in SRIOV
2) original logic always set all flags which make itself
   a dummy step

besides,
1) the ih_doorbell_range set should also be skipped
for VEGA10 SRIOV.
2) the gfx_common registers should also be skipped
for VEGA10 SRIOV.
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NEmily Deng <Emily.Deng@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Same as navi10.
Reviewed-by: NXiaojie Yuan <xiaojie.yuan@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

some subblocks of gfx fail in inject test, disable them
Signed-off-by: NDennis Li <Dennis.Li@amd.com>
Reviewed-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

check gfx error count in both ras querry function and
ras interrupt handler.

gfx ras is still disabled by default due to known stability
issue found in gpu reset.
Signed-off-by: NDennis Li <Dennis.Li@amd.com>
Reviewed-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add functions for RAS error inject and query error counter
Signed-off-by: NDennis Li <Dennis.Li@amd.com>
Reviewed-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NDennis Li <Dennis.Li@amd.com>
Reviewed-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

error injection address is not in gpu address space
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

only ue and ce errors are supported
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

add err_data parameter in interrupt cb for ras clients
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

add error data as parameter for ras interrupt cb and process it
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

query umc ras error address, translate it to gpu 4k page view
and save it.
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

add related registers, callback function and channel index table
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

more than one error address may be recorded in one query
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

remove the check of ErrorCodeExt

v2: refine the if condition for ue counting
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

create new amdgpu_umc structure to for more umc
settings in future and switch to the new structure
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

replace some 32bit macros with 64bit operations to simplify code
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

add 64 bits register access functions

v2: implement 64 bit functions in low level
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

v1: increase ras ce/ue error count
v2: log the number of correctable and uncorrectable errors
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

check umc error count in both ras querry function and
ras interrupt handler
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

init umc callback function for vega20 in sw early init phase
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Implement umc query_ras_error_count function to support querry
both correctable and uncorrectable error
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NTao Zhou <tao.zhou1@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This is common structure as UMC callback function
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

the driver needs to program RSMU and UMC registers to
support vega20 RAS feature
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

These are common structures that can be included by IP specific
source files
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NDennis Li <dennis.li@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Unused.
Acked-by: NSam Ravnborg <sam@ravnborg.org>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

And fix the fallout.
Acked-by: NSam Ravnborg <sam@ravnborg.org>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

And fix the fallout.
Acked-by: NSam Ravnborg <sam@ravnborg.org>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

And fix up the fallout.
Acked-by: NSam Ravnborg <sam@ravnborg.org>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

And fix the fallout.
Acked-by: NSam Ravnborg <sam@ravnborg.org>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

And fix the fallout.
Acked-by: NSam Ravnborg <sam@ravnborg.org>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Unused.
Acked-by: NSam Ravnborg <sam@ravnborg.org>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Unused.
Acked-by: NSam Ravnborg <sam@ravnborg.org>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

we should pass irq type, instead of irq client id,
to irq_get/put interface
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NFeifei Xu <Feifei.Xu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

VCN should be used for Vega20 later ASICs while UVD and VCE
are for previous ASICs.
Signed-off-by: NEvan Quan <evan.quan@amd.com>
Reviewed-by: NKenneth Feng <kenneth.feng@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Update for sdma ras ecc_irq and other minors.
Signed-off-by: NLe Ma <le.ma@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Arcturus VCN should powergate in the way as Navi.
Signed-off-by: NEvan Quan <evan.quan@amd.com>
Reviewed-by: NLe Ma <Le.Ma@amd.com>
Reviewed-by: NKenneth Feng <kenneth.feng@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>