Use IP versions rather than asic_type to differentiate
IP version specific features.
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

To help debugging GPUVM page faults.
Acked-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The result of an expression consisting of a single relational operator is
already of the bool type and does not need to be evaluated explicitly.

No functional change.
Signed-off-by: NZhen Lei <thunder.leizhen@huawei.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Use amdgpu_gmc_vram_pa and amdgpu_gmc_vram_cpu_pa
to simplify codes. No logic change.
Signed-off-by: NOak Zeng <Oak.Zeng@amd.com>
Signed-off-by: NHarish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

mmhub ras is only avaiable in cerntain mmhub ip
generation.
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NDennis Li <Dennis.Li@amd.com>
Reviewed-by: NJohn Clements <John.Clements@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Currently the ! operator is incorrectly being used to flip bits on
mask values. Fix this by using the bit-wise ~ operator instead.

Addresses-Coverity: ("Logical vs. bitwise operator")
Fixes: 3c9a7b7d ("drm/amdgpu: update mmhub mgcg&ls for mmhub_v2_3")
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Signed-off-by: NColin Ian King <colin.king@canonical.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Starting from vangogh, the ATCL2 and DAGB0 registers relative
to mgcg/ls has changed.

For MGCG:
Replace mmMM_ATC_L2_MISC_CG with mmMM_ATC_L2_CGTT_CLK_CTRL.

For MGLS:
Replace mmMM_ATC_L2_MISC_CG with mmMM_ATC_L2_CGTT_CLK_CTRL.
Add DAGB0_(WR/RD)_CGTT_CLK_CTRL registers.
Signed-off-by: NAaron Liu <aaron.liu@amd.com>
Acked-by: NHuang Rui <ray.huang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Starting from vangogh, the ATCL2 and DAGB0 registers relative
to mgcg/ls has changed.

For MGCG:
Replace mmMM_ATC_L2_MISC_CG with mmMM_ATC_L2_CGTT_CLK_CTRL.

For MGLS:
Replace mmMM_ATC_L2_MISC_CG with mmMM_ATC_L2_CGTT_CLK_CTRL.
Add DAGB0_(WR/RD)_CGTT_CLK_CTRL registers.
Signed-off-by: NAaron Liu <aaron.liu@amd.com>
Acked-by: NHuang Rui <ray.huang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

global noretry setting is cached to gmc.noretry
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

global noretry setting is cached to gmc.noretry
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Replace hardcoded vmid number with AMDGPU_NUM_VMID macro.
Signed-off-by: NNirmoy Das <nirmoy.das@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Just a small optimization for accessing system pages directly.
Was missed for gmc v10 since the feature landed for older gmcs
while we were still on the emulator or gmc10 and we use the AGP
aperture for zfb on the emulator.

v2: fix up the system aperture as well
Reviewed-and-tested-by: NEvan Quan <evan.quan@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Print the name of the client rather than the number.  This
makes it easier to debug what block is causing the fault.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

There are too many register offset mismatch between mmhub v2.0 and v2.3.

E.X:

mmMM_ATC_L2_MISC_CG:  0x064a(v2.0)  0x06cd(v2.3)
mmMMVM_L2_PROTECTION_FAULT_CNTL: 0x0688(v2.0) 0x0708(v2.3)
mmMMVM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32: 0x072b(v2.0) 0x0940(v2.3)
mmMMVM_CONTEXT0_PAGE_TABLE_BASE_ADDR_HI32: 0x072c(v2.0) 0x0941(v2.3)
mmMMVM_INVALIDATE_ENG0_REQ: 0x06e3(v2.0) 0x0a01(v2.3)
mmMMVM_INVALIDATE_ENG0_ACK: 0x06f5(v2.0) 0x0a02(v2.3)
mmMMVM_CONTEXT0_CNTL: 0x06c0(v2.0) 0x0740(v2.3)
mmMMVM_L2_PROTECTION_FAULT_STATUS: 0x068c(v2.0) 0x070c(v2.3)
mmMMVM_L2_PROTECTION_FAULT_CNTL: 0x0688(v2.0) 0x0708(v2.3)
mmMM_ATC_L2_MISC_CG: 0x064a(v2.0) 0x06cd(v2.3)
mmDAGB0_CNTL_MISC2: 0x0071(v2.0) 0x0096(v2.3)
...

Continuing using the same file mmhub v2.0 is not good choice, it will
introduce a lot of checking with ASIC types. And also easy to introduce the
issues that offset not align, this kind of issues are really hard to find. Van
Gogh's mmhub vm invalidation is actually caused by the offset mismatch as well.

So it would like to create a new file rather than stick to re-use orignal mmhub
v2.0 here.

v2: add missed translate_further programming.
v3: sync with latest code
v4: add missing callbacks
Signed-off-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This will allow us to have different defaults per asic
in a future patch.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NLuben Tuikov <luben.tuikov@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NEmily.Deng <Emily.Deng@amd.com>
Reviewed-by: NFrank Min <Frank.Min@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Print the name of the client rather than the number.  This
makes it easier to debug what block is causing the fault.
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add more function pointers to amdgpu_mmhub_funcs. ASIC specific
implementation of most mmhub functions are called from a general
function pointer, instead of calling different function for
different ASIC. Simplify the code by deleting duplicate functions
Signed-off-by: NOak Zeng <Oak.Zeng@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Some registers are not accessible to virtual function setup, so
skip their initialization when in VF-SRIOV mode.

v2: move SRIOV VF check into specify functions;
modify commit description and comment.
Signed-off-by: NLiu ChengZhe <ChengZhe.Liu@amd.com>
Reviewed-by: NLuben Tuikov <luben.tuikov@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Some registers are not accessible to virtual function setup, so
skip their initialization when in VF-SRIOV mode.

v2: move SRIOV VF check into specify functions;
modify commit description and comment.
Signed-off-by: NLiu ChengZhe <ChengZhe.Liu@amd.com>
Reviewed-by: NLuben Tuikov <luben.tuikov@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This patch is to move get_invalidate_req into gfxhub/mmhub level. It will avoid
mismatch of the different gfxhub/mmhub register offsets and fields in the same
gmc block.
Signed-off-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This patch is to introduce vmhub funcs helper to add following callback
(print_l2_protection_fault_status). Each GC/MMHUB register specific programming
should be in gfxhub/mmhub level.

v2: remove the condition of funcs assignment.
Signed-off-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This patch adds a member in vmhub structure to store the vm fault interrupt
masks for different version gfxhubs/mmhubs.
Signed-off-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The athub version used for navy_flounder is v2.1.
Signed-off-by: NJiansong Chen <Jiansong.Chen@amd.com>
Reviewed-by: NTao Zhou <tao.zhou1@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

navy_flounder has the same mmhub IP version with sienna_cichlid,
follow its setting.
Signed-off-by: NJiansong Chen <Jiansong.Chen@amd.com>
Reviewed-by: NTao Zhou <Tao.Zhou1@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This patch updates to use register distance member instead of hardcode
in mmhub v2.
Signed-off-by: NHuang Rui <ray.huang@amd.com>
Tested-by: NAnZhong Huang <anzhong.huang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This patch is to abstract register distances between two continuous
context domains and invalidation engines. In different ip headers, these
distances may be differences.
Signed-off-by: NHuang Rui <ray.huang@amd.com>
Tested-by: NAnZhong Huang <anzhong.huang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add gmc clockgating support for sienna_cichlid.
The athub version used for sienna_cichlid is v2.1.
Signed-off-by: NLikun Gao <Likun.Gao@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NLikun Gao <Likun.Gao@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Explicitly set the smallk size to 0 (4k).  This is the hw
default, but set it anyway just in case something else
changed it.
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

With new L1 policy, some regs are blocked at guest and they are
programed at host side. So skip programing the regs under sriov.

the regs are:
GCMC_VM_FB_LOCATION_TOP
GCMC_VM_FB_LOCATION_BASE
MMMC_VM_FB_LOCATION_TOP
MMMC_VM_FB_LOCATION_BASE
GCMC_VM_SYSTEM_APERTURE_HIGH_ADDR
GCMC_VM_SYSTEM_APERTURE_LOW_ADDR
MMMC_VM_SYSTEM_APERTURE_HIGH_ADDR
MMMC_VM_SYSTEM_APERTURE_LOW_ADDR
HDP_NONSURFACE_BASE
HDP_NONSURFACE_BASE_HI
GCMC_VM_AGP_TOP
GCMC_VM_AGP_BOT
GCMC_VM_AGP_BASE
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NTiecheng Zhou <Tiecheng.Zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Better clean that up before some automation starts to complain about it
Signed-off-by: NNirmoy Das <nirmoy.das@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

SW must acquire/release one of the vm_invalidate_eng*_sem around the
invalidation req/ack. Through this way,it can avoid losing invalidate
acknowledge state across power-gating off cycle.
To use vm_invalidate_eng*_sem, it needs to initialize
vm_invalidate_eng*_sem firstly.
Signed-off-by: Nchangzhu <Changfeng.Zhu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

SW must acquire/release one of the vm_invalidate_eng*_sem around the
invalidation req/ack. Through this way,it can avoid losing invalidate
acknowledge state across power-gating off cycle.
To use vm_invalidate_eng*_sem, it needs to initialize
vm_invalidate_eng*_sem firstly.
Signed-off-by: Nchangzhu <Changfeng.Zhu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

These were not aligned for optimal performance for GPUVM.
Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NTianci Yin <tianci.yin@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

These were not aligned for optimal performance for GPUVM.
Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NTianci Yin <tianci.yin@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The KFD code will call this function later.
Signed-off-by: NYong Zhao <Yong.Zhao@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

There is no point retrying page faults in VMID0. Those faults are
always fatal.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-and-Tested-by: NHuang Rui <ray.huang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We have done this for pre-GFX10 asics, but GFX10 did not pick up the
new change. The below is the commit message for that change.

This is recommended by HW designers. Previously when it was set to 1,
the PDE walk error in VM fault will be treated as
PERMISSION_OR_INVALID_PAGE_FAULT rather than usually expected OTHER_FAULT.
As a result, the retry control in VM_CONTEXT*_CNTL will change accordingly.

The above behavior is kind of abnormal. Furthermore, the
PDE_FAULT_CLASSIFICATION == 1 feature was targeted for very old ASICs
and it never made it way to production. Therefore, we should set it to 0.
Signed-off-by: NYong Zhao <Yong.Zhao@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

add navi12 define
Signed-off-by: NXiaojie Yuan <xiaojie.yuan@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>