radeon must load only after amdkfd's loading has been completed. If that
is not enforced, then radeon's call into amdkfd's functions will cause a
kernel BUG.

When radeon and amdkfd are built as kernel modules, that rule is
enforced by the kernel's modules loading mechanism. When radeon and
amdkfd are built inside the kernel image, that rule is enforced by
ordering in the drm Makefile (amdkfd before radeon).

Instead of using drm Makefile ordering, we can now use deferred
loading as amdkfd now returns -EPROBE_DEFER in kgd2kfd_init() when it is
not yet loaded.

This patch defers radeon loading by propagating -EPROBE_DEFER to the
kernel's drivers loading infrastructure. That will put radeon into the
pending drivers list (see description in dd.c). Once amdkfd is loaded,
a call to kgd2kfd_init() will return successfully and radeon will be
able to load.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

Current dependencies between amdkfd and radeon/amdgpu force the loading
of amdkfd _before_ radeon and/or amdgpu are loaded. When all these kernel
drivers are built as modules, this ordering is enforced by the kernel
built-in mechanism of loading dependent modules.

However, there is no such mechanism in case where all these drivers are
compiled inside the kernel image (not as modules). The current way to
enforce loading of amdkfd before radeon/amdgpu, is to put amdkfd before
radeon/amdgpu in the drm Makefile, but that method is way too fragile.

In addition, there is no kernel mechanism to check whether a kernel
driver that is built inside the kernel image, has already been loaded.

To solve this, this patch adds to kfd_module.c a new static variable,
amdkfd_init_completed, that is set to 1 only when amdkfd's
module initialization function has been completed (successfully).

kgd2kfd_init(), which is the initialization function of the
kgd-->kfd interface, and which is the first function in amdkfd called by
radeon/amdgpu, will return successfully only if amdkfd_init_completed is
equal 1.

If amdkfd_init_completed is not equal to 1, kgd2kfd_init() will
return -EPROBE_DEFER to signal radeon/amdgpu they need to defer
their loading until amdkfd is loaded.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch splits the KGD_ENGINE_SDMA to KGD_ENGINE_SDMA1 and
KGD_ENGINE_SDMA2 to match CZ definitions.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds three new interfaces to kfd2kgd interface file of radeon.

The interfaces are:

- Check if a specific VMID has a valid PASID mapping
- Retrieve the PASID which is mapped to a specific VMID
- Issue a VMID invalidation request to the ATC
Signed-off-by: NAlexey Skidanov <Alexey.Skidanov@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds new interface functions to the kfd2kgd interface file. The
new functions allow to perform H/W debugger operations by writing to GPU
registers.
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds a new interface function to the kfd->kgd interface.
The function is kgd_init_interrupts() and its function is to
initialize a pipe's interrupts.

The function currently enables the timestamp interrupt and the
bad opcode interrupt.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The current code can only support one kgd instance. We have to
support multiple kgd instances in one system. i.e two amdgpu or two
radeon or one amdgpu + one radeon or more than two kgd instances.
Signed-off-by: NXihan Zhang <xihan.zhang@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

The current CP firmware can handle Usermode Queues only on MEC1.
To reflect this firmware change, this commit reduces number of compute pipelines
to 4 - 1, from 8 - 1 (the first pipeline is allocated for kgd).
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Cc: stable@vger.kernel.org

This patch fixes the behavior of kgd_init_pipeline in that this function
shouldn't automatically increase the pipe_id argument by 1 right at the start
of the function.

This is because the first_pipe value might not be always 1, and because a
proper interface function should not hide this info inside its implementation.
In other words, the calling function should provide the real pipe_id and not
count on kgd_init_pipeline to "fix" it.
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlexey Skidanov <Alexey.skidanov@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch adds the implementation of the gtt interface functions.

The allocate function will allocate a single bo, pin and map it to kernel
memory. It will return the gpu address and cpu ptr as arguments.

v2:

The bulk of the allocations in the GART is for MQDs. MQDs represent active
user-mode queues, which are on the current runlist. It is important to
remember that active queues doesn't necessarily mean scheduled/running
queues, especially if there is over-subscription of queues or more than a
single HSA process.

Because the scheduling of the user-mode queues is done by the CP firmware,
amdkfd doesn't have any indication if the queue is scheduled or not. If the
CP will try to schedule a queue, and its MQD is not present, this will
probably stuck the CP permanently, as it will load garbage from the GART
(the address of the MQD is given to the CP inside the runlist packet).

In addition, there are a couple of small allocations which also should
always be pinned - runlist packets (2 packets) and HPDs. runlist packets can
be quite large, depending on number of processes and queues.

This new allocate function represents the short/mid-term solution of limiting
the total memory consumption to around 4MB by default.

The long-term solution is to create a mechanism through which radeon/ttm can
ask amdkfd to clear GART/VRAM memory due to memory pressure.
Then, amdkfd will preempt the running queues and wait until the memory pressure
is over. After that, amdkfd will reschedule the queues.
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlexey Skidanov <Alexey.skidanov@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch implements the new SDMA interface functions. It also adds defines
and structures related to SDMA registers.

v2: Removed init_sdma_engines() from interface. Initialization is done in
radeon.

v3:
- Removed unused defines.
- Added SDMA_ prefix to defines that didn't have them.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch changes the radeon_kfd_init(), which is used to initialize the
interface between radeon and amdkfd, so the interface will be initialized only
if amdkfd was build, either as module or inside the kernel image.

In the modules case, the symbol_request() will be used (same as old code). In
the in-image compilation case, a direct call to kgd2kfd_init() will be done.
For other cases, radeon_kfd_init() will just return false.

This patch is necessary because in case of the following specific
configuration: kernel 32-bit, no modules support, random kernel base and no
hibernation, the symbol_request() doesn't work as expected - it doesn't return
NULL if the symbol doesn't exists - which makes the kernel panic.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch fixes a bug in kgd_set_pasid_vmid_mapping(), where the function
only updated the ATC registers (IOMMU) with the new VMID <--> PASID mapping,
but didn't update the IH (Interrupt) registers.

The bug only occurs when using non-HWS mode. In HWS mode, the CP automatically
does the VMID <--> PASID mapping.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>

If timeout happens, kgd_hqd_destroy() just returns -ETIME leaving queue
acquired.
It may cause a deadlock, so the patch proposes to release queue before return.

Found by Linux Driver Verification project (linuxtesting.org).
Signed-off-by: NAlexey Khoroshilov <khoroshilov@ispras.ru>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch implements a new interface that was added to the kfd-->kgd interface.
The new interface function retrieves the firmware version that is currently
in use by a specific engine. The firmware was uploaded to the engine by the
radeon driver.

v2: Returns the fw version of the specific engine, as passed into the function
by a new parameter
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch moves to radeon the initialization of compute vmid.

That initializations was done in kfd-->kgd interface, but doing it in radeon
as part of radeon's H/W initialization routines is more appropriate.

In addition, this simplifies the kfd-->kgd interface.

The patch removes the function from the interface file and from the interface
declaration file.

The function initializes memory apertures to fixed base/limit address and non
cached memory types.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch adds the interface between the radeon driver and the amdkfd driver.
The interface implementation is contained in radeon_kfd.c and radeon_kfd.h.

The interface itself is represented by a pointer to struct
kfd_dev. The pointer is located inside radeon_device structure.

All the register accesses that amdkfd need are done using this interface. This
allows us to avoid direct register accesses in amdkfd proper,  while also
avoiding locking between amdkfd and radeon.

The single exception is the doorbells that are used in both of the drivers.
However, because they are located in separate pci bar pages, the danger of
sharing registers between the drivers is minimal.

Having said that, we are planning to move the doorbells as well to radeon.

v3:

Add interface for sa manager init and fini. The init function will allocate a
buffer on system memory and pin it to the GART address space via the radeon sa
manager.

All mappings of buffers to GART address space are done via the radeon sa
manager. The interface of allocate memory will use the radeon sa manager to sub
allocate from the single buffer that was allocated during the init function.

Change lower_32/upper_32 calls to use linux macros

Add documentation for the interface

v4:

Change ptr field type in kgd_mem from uint32_t* to void* to match to type that
is returned by radeon_sa_bo_cpu_addr

v5:

Change format of mqd structure to work with latest KV firmware
Add support for AQL queues creation to enable working with open-source HSA
runtime.
Move generic kfd-->kgd interface and other generic kgd definitions to a generic
header file that will be used by AMD's radeon and amdgpu drivers
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>