Current dependencies between amdkfd and radeon/amdgpu force the loading
of amdkfd _before_ radeon and/or amdgpu are loaded. When all these kernel
drivers are built as modules, this ordering is enforced by the kernel
built-in mechanism of loading dependent modules.

However, there is no such mechanism in case where all these drivers are
compiled inside the kernel image (not as modules). The current way to
enforce loading of amdkfd before radeon/amdgpu, is to put amdkfd before
radeon/amdgpu in the drm Makefile, but that method is way too fragile.

In addition, there is no kernel mechanism to check whether a kernel
driver that is built inside the kernel image, has already been loaded.

To solve this, this patch adds to kfd_module.c a new static variable,
amdkfd_init_completed, that is set to 1 only when amdkfd's
module initialization function has been completed (successfully).

kgd2kfd_init(), which is the initialization function of the
kgd-->kfd interface, and which is the first function in amdkfd called by
radeon/amdgpu, will return successfully only if amdkfd_init_completed is
equal 1.

If amdkfd_init_completed is not equal to 1, kgd2kfd_init() will
return -EPROBE_DEFER to signal radeon/amdgpu they need to defer
their loading until amdkfd is loaded.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds a new kernel module parameter to amdkfd,
called send_sigterm.

This parameter specifies whether amdkfd should send the
SIGTERM signal to an HSA process, when the following conditions
occur:

1. The GPU triggers an exception regarding a kernel that was
   issued by this process.

2. The HSA process isn't waiting on an event that handles
   this exception.

The default behavior is not to send a SIGTERM and suffice
with a dmesg error print.
Reviewed-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The current code can only support one kgd instance. We have to
support multiple kgd instances in one system. i.e two amdgpu or two
radeon or one amdgpu + one radeon or more than two kgd instances.
Signed-off-by: NXihan Zhang <xihan.zhang@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch replaces the two current amdkfd module parameters with a new one.

The current parameters that are being replaced are:

- Maximum number of HSA processes
- Maximum number of queues per process

The new parameter that replaces them is called "Maximum queues per device"

This replacement achieves two goals:

- Allows the user to have as many HSA processes as it wants (until
  a maximum of 512 HSA processes in Kaveri).

- Removes the limitation the user had on maximum number of queues per HSA
  process. E.g. the user can now have processes which only have one queue and
  other processes which have hundreds of queues, while before the user
  couldn't have more than 128 queues per process (as default).

The default value of the new parameter is 4096 (32 * 128, which were the
defaults of the old parameters). There is almost no additional GART memory
required for the default case. As a reminder, this amount of queues requires a
little bit below 4MB of GART memory.

v2:
In addition, This patch defines a new counter for queues accounting in the DQM
structure. This is done because the current counter only counts active queues
which allows the user to create more queues than the
max_num_of_queues_per_device module parameter allows.

However, we need the current counter for the runlist packet build process, so
the solution is to have a dedicated counter for this accounting.
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NBen Goz <ben.goz@amd.com>

Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch adds a new parameter to the amdkfd driver. This parameter enables
the user to select the scheduling policy of the CP. The choices are:

* CP Scheduling with support for over-subscription
* CP Scheduling without support for over-subscription
* Without CP Scheduling

Note that the third option (Without CP scheduling) is only for debug purposes
and bringup of new H/W. As such, it is _not_ guaranteed to work at all times on
all H/W versions.

v3: Fixed description of parameter, changed the permissions to read_only, added
a verification of the value and added documentation

v5: Set default sched_policy to HWS as it is now supported by firmware
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch adds the process module and three helper modules:

- kfd_process, which handles process which open /dev/kfd

- kfd_doorbell, which provides helper functions for doorbell allocation,
  release and mapping to userspace

- kfd_pasid, which provides helper functions for pasid allocation and release

- kfd_aperture, which provides helper functions for managing the LDS, Local GPU
  memory and Scratch memory apertures of the process

This patch only contains the basic kfd_process module, which doesn't contain
the reference to the queue scheduler. This was done to allow easier code review.

Also, this patch doesn't contain the calls to the IOMMU driver for binding the
pasid to the device. Again, this was done to allow easier code review

The kfd_process object is created when a process opens /dev/kfd and is closed
when the mm_struct of that process is teared-down.

v3:

Removed kfd_vidmem.c file
Replaced direct mmput call to mmu_notifier release
Removed typedefs
Moved bool field to end of the structure
Added new kernel params for gart usage limitation
Added initialization of sa manager
Fixed debug messages
Remove support for LDS in 32 bit
Changed code to support mmap of doorbell pages from userspace
Added documentation for apertures

v4: Replaced RCU by SRCU for kfd_process list management

v5:

Move amdkfd from drm/radeon/ to drm/amd/
Rename kfd_aperture.c to kfd_flat_memory.c
Protect against multiple init calls
MQD size is H/W dependent so moved it to device info structure
Rename kfd_mem_obj structure's members
Use delayed function for process tear-down
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch adds the topology module to the driver. The topology is exposed to
userspace through the sysfs.

The calls to add and remove a device to/from topology are done by the radeon
driver.

v3:

The CPU information, that is provided in the topology section of the amdkfd
driver, is extracted from the CRAT table. Unlike the CPU information located
in /sys/devices/system/cpu/cpu*, which is extracted from the SRAT table.

While the CPU information provided by the CRAT and the SRAT tables might be
identical, the node topology might be different. The SRAT table contains the
topology of CPU nodes only. The CRAT table contains the topology of CPU and GPU
nodes together (and can be interleaved). For example CPU node 1 in SRAT can be
CPU node 3 in CRAT. Furthermore it's worth to mention that the CRAT table
contains only HSA compatible nodes (nodes which are compliant with the HSA
spec).

To recap, amdkfd exposes a different kind of topology than the one exposed by
/sys/devices/system/cpu/cpu even though it may contain similar information.

v4:

The topology module doesn't support uevent handling and doesn't notify the
userspace about runtime modifications. It is up to the userspace to acquire
snapshots of the topology information created by the amdkfd and exposed
in sysfs.

The following is an example of how the topology looks on a Kaveri A10-7850K
system with amdkfd installed:

/sys/devices/virtual/kfd/kfd/
|
--- topology/
      |
      |--- generation_id
      |--- system_properties
      |--- nodes/
            |
            |--- 0/
                 |
                 |--- gpu_id
                 |--- name
                 |--- properties
                 |--- caches/
                      |
                      |--- 0/
                           |
                           |--- properties
                      |--- 1/
                           |
                           |--- properties
                      |--- 2/
                           |
                           |--- properties
                 |--- io_links/
                      |
                 |--- mem_banks/
                      |
                      |--- 0/
                           |
                           |--- properties
                      |--- 1/
                           |
                           |--- properties
                      |--- 2/
                           |
                           |--- properties
                      |--- 3/
                           |
                           |--- properties

v5:

Move amdkfd from drm/radeon/ to drm/amd/

Add a check if dev->gpu pointer is null before accessing it in the
node_show function in kfd_topology.c
This situation may occur when amdkfd is loaded and there is a GPU with a CRAT
table, but that GPU isn't supported by amdkfd
Signed-off-by: NEvgeny Pinchuk <evgeny.pinchuk@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch adds the amdkfd skeleton driver. The driver does nothing except
define a /dev/kfd device.

It returns -ENODEV on all amdkfd IOCTLs.

v3: Move bool field to the end of structure, removed the pmc ioctls and added
a meaningful error message for ioctl error.

v5:

Create a new folder drm/amd and move amdkfd from drm/radeon/ to drm/amd/
Remove scheduler_class from kfd_priv.h as it was never used
Add skeleton implementation of the Get Version IOCTL

v6:
Update module version to the correct number and remove the "default m" from the
Kconfig file
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>