Signed-off-by: NTom St Denis <tom.stdenis@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NTom St Denis <tom.stdenis@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NTom St Denis <tom.stdenis@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NTom St Denis <tom.stdenis@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NTom St Denis <tom.stdenis@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NTom St Denis <tom.stdenis@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

To allocate additional space for the dynamic cu masks.
Confirmed with the hw team that we only need 1 dword
for the mask.  The mask is the same for each SE so
you only need 1 dword.
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

mmVGT_INDEX_TYPE has no default value, need to make sure
it's initialized when gfx is initialized.
Signed-off-by: NKen Wang <Ken.Wang@amd.com>
Reviewed-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We won't use this member in other files, so set it static.
Signed-off-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Saves us even more loc.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Saves us quite a bunch of loc.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NEvan Quan <evan.quan@amd.com>
Reviewed-by: NKen Wang <ken.wang@amd.com>
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Copy the approach taken by gfx8, which simplifies the code, and set the
instance index properly. The latter is required for debugging, e.g. for
reading wave status by UMR.
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Copy the approach taken by gfx8, which simplifies the code, and set the
instance index properly. The latter is required for debugging, e.g. for
reading wave status by UMR.
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This is just future proofing code, not something that can be triggered
in real life.  We're testing to make sure we don't shift wrap when we
do "1ull << i" so "i" has to be in the 0-63 range.  If it's 64 then we
have gone too far.
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Drop the KCQ disabling via KIQ.  We disable the MEC shortly
after anyway, so there is no need to wait for all of this.
Doing so seems to leave the MEC in a bad way.
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This is ported from gfx8.
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Newer asics with 4 SEs are not able to fit the entire bitmask in the
original field, use an array instead.

v2: keep cu_ao_mask for backward compatibility.
Signed-off-by: NFlora Cui <Flora.Cui@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NFengguang Wu <fengguang.wu@intel.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This got lost when the code was revamped.  Copy/paste bug from
gfx8.
Reported-by: NEvan Quan <evan.quan@amd.com>
Fixes: 78c16834 (drm/amdgpu: allow split of queues with kfd at queue granularity v4)
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

A couple of simple tidy ups to register programming.
Signed-off-by: NTom St Denis <tom.stdenis@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

(v2): Avoid using 'data' uninitialized
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

No need to clear it.  The values are set explicitly.
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The interrupt registers are not indexed.

Fixes: 763a47b8 (drm/amdgpu: teach amdgpu how to enable interrupts for any pipe v3)
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

It was duplicated across multiple generations.
Reviewed-by: NAlex Xie <AlexBin.Xie@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This will allow us to share more mec code.
Reviewed-by: NAlex Xie <AlexBin.Xie@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Lots more common stuff.
Reviewed-by: NAlex Xie <AlexBin.Xie@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

They are gfx related, not general helpers.
Reviewed-by: NAlex Xie <AlexBin.Xie@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This overrode what queue was actually assigned for kiq.
Reviewed-by: NAlex Xie <AlexBin.Xie@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This was missed when Andres' queue patches were rebased.

Fixes: 42794b27 (drm/amdgpu: take ownership of per-pipe configuration v3)
Reviewed-by: NAlex Xie <AlexBin.Xie@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Same function was duplicated in all gfx IP files.
Reviewed-by: NAlex Xie <AlexBin.Xie@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Fixes hangs on single MEC asics.

Fixes: 2ed286fb434 (drm/amdgpu: new queue policy, take first 2 queues of each pipe v2)
Reviewed-by: NAlex Xie <AlexBin.Xie@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The same function was duplicated in all the gfx IPs. Use
a single implementation for all.

v2: use static inline (Alex Xie)
Reviewed-by: NAlex Xie <AlexBin.Xie@amd.com>
Suggested-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Instead of taking the first pipe and giving the rest to kfd, take the
first 2 queues of each pipe.

Effectively, amdgpu and amdkfd own the same number of queues. But
because the queues are spread over multiple pipes the hardware will be
able to better handle concurrent compute workloads.

amdgpu goes from 1 pipe to 4 pipes, i.e. from 1 compute threads to 4
amdkfd goes from 3 pipe to 4 pipes, i.e. from 3 compute threads to 4

gfx9 was missed when this patch set was rebased to include gfx9.
Acked-by: NTom St Denis <tom.stdenis@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Pipes provide better concurrency than queues, therefore we want to make
sure that apps use queues from different pipes whenever possible.

Optimize for the trivial case where an app will consume rings in order,
therefore we don't want adjacent rings to belong to the same pipe.

gfx9 was missed when these patches were rebased.
Reviewed-by: NTom St Denis <tom.stdenis@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Instead of picking an arbitrary queue for KIQ, search for one according
to policy. The queue must be unused.

Also report the KIQ as an unavailable resource to KFD.

In testing I ran into KCQ initialization issues when using pipes 2/3 of
MEC2 for the KIQ. Therefore the policy disallows grabbing one of these.

v2: fix (ring.me + 1) to (ring.me -1) in amdgpu_amdkfd_device_init
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The assumption that we are only using the first pipe no longer holds.
Instead, calculate the queue_mask from the queue_bitmap.
Acked-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The current implementation is hardcoded to enable ME1/PIPE0 interrupts
only.

This patch allows amdgpu to enable interrupts for any pipe of ME1.

v2: added gfx9 support
v3: use soc15_grbm_select for gfx9
Acked-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Previously the queue/pipe split with kfd operated with pipe
granularity. This patch allows amdgpu to take ownership of an arbitrary
set of queues.

It also consolidates the last few magic numbers in the compute
initialization process into mec_init.

v2: support for gfx9
v3: renamed AMDGPU_MAX_QUEUES to AMDGPU_MAX_COMPUTE_QUEUES
v4: fix off-by-one in num_mec checks in *_compute_queue_acquire
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Make amdgpu the owner of all per-pipe state of the HQDs.

This change will allow us to split the queues between kfd and amdgpu
with a queue granularity instead of pipe granularity.

This patch fixes kfd allocating an HDP_EOP region for its 3 pipes which
goes unused.

v2: support for gfx9
v3: fix gfx7 HPD intitialization
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>