Various bug fixes and improvements that accumulated over the last two
years.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

In most cases, BUG_ONs can be replaced with WARN_ON with an error
return. In some void functions just turn them into a WARN_ON and
possibly an early exit.

v2:
* Cleaned up error handling in pm_send_unmap_queue
* Removed redundant WARN_ON in kfd_process_destroy_delayed
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Remove BUG_ONs that check for NULL pointer arguments that are
dereferenced in the same function. Dereferencing the NULL pointer
will generate a BUG anyway, so the explicit check is redundant and
unnecessary overhead.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

See https://kernel.org/doc/html/latest/process/coding-style.html
under "14) Allocating Memory" for rationale behind removing the
x=alloc(sizeof(struct) style and using x=alloc(sizeof(*x) instead
Signed-off-by: NKent Russell <kent.russell@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Remove gotos that do not feature any common cleanup, and use gotos
instead of repeating cleanup commands.

According to kernel.org: "The goto statement comes in handy when a
function exits from multiple locations and some common work such as
cleanup has to be done. If there is no cleanup needed then just return
directly."

v2: Applied review suggestions in create_queue_nocpsch
Signed-off-by: NKent Russell <kent.russell@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Upstream prefers the !x notation to x==NULL or x==false. Along those lines
change the ==true or !=NULL references as well. Also make the references
to !x the same, excluding () for readability.
Signed-off-by: NKent Russell <kent.russell@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Consolidate log commands so that dev_info(NULL, "Error...") uses the more
accurate pr_err, remove the module name from the log (can be seen via
dynamic debugging with +m), and the function name (can be seen via
dynamic debugging with +f). We also don't need debug messages saying
what function we're in. Those can be added by devs when needed

Don't print vendor and device ID in error messages. They are typically
the same for all GPUs in a multi-GPU system. So this doesn't add any
value to the message.

Lastly, remove parentheses around %d, %i and 0x%llX.
According to kernel.org:
"Printing numbers in parentheses (%d) adds no value and should be
avoided."
Signed-off-by: NKent Russell <kent.russell@amd.com>
Signed-off-by: NYong Zhao <Yong.Zhao@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Using checkpatch.pl -f <file> showed a number of style issues. This
patch addresses as many of them as possible. Some long lines have been
left for readability, but attempts to minimize them have been made.

v2: Broke long lines in gfx_v7 get_fw_version
Signed-off-by: NKent Russell <kent.russell@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Use shared_resources.queue_bitmap to determine the queues available
for KFD in each pipe.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This is just future proofing code, not something that can be triggered
in real life.  We're testing to make sure we don't shift wrap when we
do "1ull << i" so "i" has to be in the 0-63 range.  If it's 64 then we
have gone too far.
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Dead code.

Change-Id: Ic0bb1bcca87e96bc5e8fa9894727b0de152e8818
Signed-off-by: NJay Cornwall <Jay.Cornwall@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Update the KGD to KFD interface to allow sharing pipes with queue
granularity instead of pipe granularity.

This allows for more interesting pipe/queue splits.

v2: fix overflow check for res.queue_mask
v3: fix shift overflow when setting res.queue_mask
v4: fix comment in is_pipeline_enabled()
v5: clamp res.queue_mask to the first MEC only
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Make amdgpu the owner of all per-pipe state of the HQDs.

This change will allow us to split the queues between kfd and amdgpu
with a queue granularity instead of pipe granularity.

This patch fixes kfd allocating an HDP_EOP region for its 3 pipes which
goes unused.

v2: support for gfx9
v3: fix gfx7 HPD intitialization
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Found-By: Coccinelle
Signed-off-by: NEdward O'Callaghan <eocallaghan@alterapraxis.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This commit moves the reset wavefront flag to per process per device
data structure, so we can support multiple devices.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This commit makes sure that on process termination, after
we're destroying all the active queues, we're killing all the
existing wave front of the current process.

By doing this we're making sure that if any of the CUs were blocked
by infinite loop we're enforcing it to end the shader explicitly.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The wave control operation supports several command types executed upon
existing wave fronts that belong to the currently debugged process.

The available commands are:

HALT   - Freeze wave front(s) execution
RESUME - Resume freezed wave front(s) execution
KILL   - Kill existing wave front(s)
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds support for static user-mode queues in QCM.
Queues which are designated as static can NOT be preempted by
the CP microcode when it is executing its scheduling algorithm.

This is needed for supporting the debugger feature, because we
can't allow the CP to preempt queues which are currently being debugged.

The number of queues that can be designated as static is limited by the
number of HQDs (Hardware Queue Descriptors).
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds the interrupt handling module, kfd_interrupt.c, and its
related members in different data structures to the amdkfd driver.

The amdkfd interrupt module maintains an internal interrupt ring
per amdkfd device. The internal interrupt ring contains interrupts
that needs further handling. The extra handling is deferred to
a later time through a workqueue.

There's no acknowledgment for the interrupts we use. The hardware
simply queues a new interrupt each time without waiting.

The fixed-size internal queue means that it's possible for us to lose
interrupts because we have no back-pressure to the hardware.

However, only interrupts that are "wanted" by amdkfd, are copied into
the amdkfd s/w interrupt ring, in order to minimize the chances
for overflow of the ring.
Signed-off-by: NAndrew Lewycky <Andrew.Lewycky@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch fixes a bug where sdma vm wasn't initialized when
an sdma queue was created in HWS mode.

This caused GPUVM faults to appear on dmesg and it is one of the
causes that SDMA queues are not working.
Signed-off-by: NXihan Zhang <xihan.zhang@amd.com>
Reviewed-by: NBen Goz <ben.goz@amd.comt>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>
Cc: stable@vger.kernel.org
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

Sometimes we might unregister process that have queues, because we couldn't
preempt the queues. Until now we blocked it with BUG_ON but instead just
print it as debug.
Reviewed-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>
Cc: stable@vger.kernel.org
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

The current code can only support one kgd instance. We have to
support multiple kgd instances in one system. i.e two amdgpu or two
radeon or one amdgpu + one radeon or more than two kgd instances.
Signed-off-by: NXihan Zhang <xihan.zhang@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

fence_wait_timeout() is an exported kernel symbol, so we should rename our
local function to something different.
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch fixes the SDMA queue initialization, when running in non-HWS mode.

The first fix is to move the initialization of SDMA VM parameters before the
initialization of the SDMA MQD.

The second fix is to load the MQD to an HQD after the initialization of the MQD.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

get_pipes_num() calls BUG_ON so we can't set it as inline because it produces a
warning as BUG_ON() uses static variables when it is expanded.
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch fixes a bug in the initialization of the pipelines. The
init_pipelines() function was called with a constant value of 0 in the
first_pipe argument. This is an error because amdkfd doesn't handle pipe 0.

The correct way is to pass the value that get_first_pipe() returns as the
argument for first_pipe.

This bug appeared in 3.19 (first version with amdkfd) and it causes around 15%
drop in CPU performance of Kaveri (A10-7850).

v2: Don't set get_first_pipe() as inline because it calls BUG_ON()
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Cc: stable@vger.kernel.org
Tested-by: NMichel Dänzer <michel.daenzer@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch fixes a bug where the first_pipe index passed into init_pipelines()
was a #define instead of the value that is passed into amdkfd by radeon
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>

This patch handles a case where amdkfd tries to destroy a queue but the queue
type is invalid.
This case occurs in non-HWS path.
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>

This patch fixes a bug when calling to init_pipeline() interface.
The index that was passed to that function didn't take into account the
first_pipe value, which represents the first pipe index that is under amdkfd's
responsibility.
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>

dqm->queue_count tracks queues in the active state only. In a few
places this count is modified unconditionally, leading to an incorrect
value when the UPDATE_QUEUE ioctl is used to make a queue inactive.
Signed-off-by: NJay Cornwall <jay.cornwall@amd.com>
Reviewed-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch replaces the two current amdkfd module parameters with a new one.

The current parameters that are being replaced are:

- Maximum number of HSA processes
- Maximum number of queues per process

The new parameter that replaces them is called "Maximum queues per device"

This replacement achieves two goals:

- Allows the user to have as many HSA processes as it wants (until
  a maximum of 512 HSA processes in Kaveri).

- Removes the limitation the user had on maximum number of queues per HSA
  process. E.g. the user can now have processes which only have one queue and
  other processes which have hundreds of queues, while before the user
  couldn't have more than 128 queues per process (as default).

The default value of the new parameter is 4096 (32 * 128, which were the
defaults of the old parameters). There is almost no additional GART memory
required for the default case. As a reminder, this amount of queues requires a
little bit below 4MB of GART memory.

v2:
In addition, This patch defines a new counter for queues accounting in the DQM
structure. This is done because the current counter only counts active queues
which allows the user to create more queues than the
max_num_of_queues_per_device module parameter allows.

However, we need the current counter for the runlist packet build process, so
the solution is to have a dedicated counter for this accounting.
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NBen Goz <ben.goz@amd.com>

In order not to occupy the current core and thus prevent the core from
servicing IOMMU PPR requests, this patch replaces the call in DQM to
cpu_relax() with a call to schedule().
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>

This patch fixes a minor bug in allocate_hqd(), where the loop run from the
next-to-allocate pipe until the number of pipes.

This is wrong because we need to consider the possibility where
next-to-allocate pipe is not 0, and thus, the for-loop only checks part of the
pipes and doesn't wrap-around, as it supposed to do.

Therefore, we add another counting variable to make sure we go over all the
pipes, regardless of where we start to look at the first iteration of the loop.

This bug only affected non-HWS mode. In HWS mode, the CP fw is responsible for
allocating the HQD.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>

This patch change the calls throughout the amdkfd driver from the old kfd-->kgd
interface to the new kfd gtt sa inside amdkfd

v2: change the new call in sdma code that appeared because of the sdma feature
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlexey Skidanov <Alexey.skidanov@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch adds support for SDMA user-mode queues to the QCM - the Queue
management system that manages queues-per-device and queues-per-process.

v2: Remove calls to interface function that initializes sdma engines.

v3: Use the new names of some of the defines.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>