Current dependencies between amdkfd and radeon/amdgpu force the loading
of amdkfd _before_ radeon and/or amdgpu are loaded. When all these kernel
drivers are built as modules, this ordering is enforced by the kernel
built-in mechanism of loading dependent modules.

However, there is no such mechanism in case where all these drivers are
compiled inside the kernel image (not as modules). The current way to
enforce loading of amdkfd before radeon/amdgpu, is to put amdkfd before
radeon/amdgpu in the drm Makefile, but that method is way too fragile.

In addition, there is no kernel mechanism to check whether a kernel
driver that is built inside the kernel image, has already been loaded.

To solve this, this patch adds to kfd_module.c a new static variable,
amdkfd_init_completed, that is set to 1 only when amdkfd's
module initialization function has been completed (successfully).

kgd2kfd_init(), which is the initialization function of the
kgd-->kfd interface, and which is the first function in amdkfd called by
radeon/amdgpu, will return successfully only if amdkfd_init_completed is
equal 1.

If amdkfd_init_completed is not equal to 1, kgd2kfd_init() will
return -EPROBE_DEFER to signal radeon/amdgpu they need to defer
their loading until amdkfd is loaded.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

Remove an unnecassary cast in the argument to kfree.

Found using Coccinelle. The semantic patch used to find this is as follows:

//<smpl>
@@
type T;
expression *f;
@@

- kfree((T *)(f));
+ kfree(f);
//</smpl>
Signed-off-by: NAmitoj Kaur Chawla <amitoj1606@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

8f1d57c1 ("amdkfd: don't open-code memdup_user()") mistakenly uses
an uninitialized local pointer, gcc complains:

  drivers/gpu/drm/amd/amdkfd/kfd_chardev.c: In function ‘kfd_ioctl_dbg_address_watch’:
  drivers/gpu/drm/amd/amdkfd/kfd_chardev.c:562:12: warning: ‘args_buff’ may be used uninitialized in this function [-Wmaybe-uninitialized]
    args_buff = memdup_user(args_buff,
                ^

Fix it.
Signed-off-by: NBorislav Petkov <bp@suse.de>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: NAl Viro <viro@zeniv.linux.org.uk>

Signed-off-by: NAl Viro <viro@zeniv.linux.org.uk>

The latter is a default version of <asm/mman.h> and not for driver use.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

A logical AND operation was used during mask and shift, instead of a
bitwise AND operation. This patch fixes this bug by changing the
operation to bitwise AND.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NBen Goz <ben.goz@amd.com>
Reviewed-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch makes use of the new amd headers (that are part of the new
amdgpu driver), instead of private defines.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The MAP_QUEUES packet length for Carrizo is different than for Kaveri.
Therefore, we now need to calculate the runlist length with regard to the
underlying H/W.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds support for the VI APU in the DQM module.

Most of the functionality of DQM is shared between CI and VI. Therefore,
only a handful of functions are required to be in the
H/W-specific part of DQM.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch implements all the VI MQD manager functions.
This is done in a different file as the MQD format is different
between CI and VI
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds the PCI IDs of supported CZ devices to the
supported_devices structure in amdkfd. That structure is used during the
amdkfd probing stage, to check if the currently probed device is eligible
to be handled by amdkfd.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Currently pdd is validate after dereferencing it, which is
not correct, Thus validate pdd before its first use.
Signed-off-by: NManinder Singh <maninder1.s@samsung.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

buf_size_in_bytes must be large enough to hold ->num_watch_points and
watch_mode so I have added a sizeof(int) * 2 to the minimum size.

Also we have to subtract sizeof(*args) from the max args_idx limit so
that it matches the allocation.  Also I changed a > to >= for the last
compare.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds two missing properties initializations to the device
info structure of CZ.

As we don't have CZ support yet, it isn't critical, but its important to
fix this now instead of forgetting about it later.
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This commit moves the reset wavefront flag to per process per device
data structure, so we can support multiple devices.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The CONFIG_ prefix is reserved for Kconfig options in Make and CPP
syntax.  Various static analysis tools rely on this naming convention
and check if CONFIG_ prefixed symbols are defined Kconfig.  Hence add
yet another prefix AMD_ to CONFIG_REG_{BASE,END,SISE} to apply to this
convention and make static analysis tools happy.
Signed-off-by: NValentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch fixes a bug where the number of watch points
was shown before it was actually calculated
Signed-off-by: NAlexey Skidanov <Alexey.Skidanov@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This commit makes sure that on process termination, after
we're destroying all the active queues, we're killing all the
existing wave front of the current process.

By doing this we're making sure that if any of the CUs were blocked
by infinite loop we're enforcing it to end the shader explicitly.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

v2:

- rename get_dbgmgr_mutex to kfd_get_dbgmgr_mutex to namespace it
- change void* to uint64_t inside ioctl arguments
- use kmalloc instead of kzalloc because we use copy_from_user
  immediately after it
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

v2:

- rename get_dbgmgr_mutex to kfd_get_dbgmgr_mutex to namespace it
- change void* to uint64_t inside ioctl arguments
- use kmalloc instead of kzalloc because we use copy_from_user
  immediately after it
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

v2: rename get_dbgmgr_mutex to kfd_get_dbgmgr_mutex to namespace it
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The address watch operation gives the ability to specify watch points
which will generate a shader breakpoint, based on a specified single
address or range of addresses.

There is support for read/write/any access modes.
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The wave control operation supports several command types executed upon
existing wave fronts that belong to the currently debugged process.

The available commands are:

HALT   - Freeze wave front(s) execution
RESUME - Resume freezed wave front(s) execution
KILL   - Kill existing wave front(s)
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds the skeleton H/W debugger module support. This code
enables registration and unregistration of a single HSA process at a
time.

The module saves the process's pasid and use it to verify that only the
registered process is allowed to execute debugger operations through the
kernel driver.

v2: rename get_dbgmgr_mutex to kfd_get_dbgmgr_mutex to namespace it
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds support for static user-mode queues in QCM.
Queues which are designated as static can NOT be preempted by
the CP microcode when it is executing its scheduling algorithm.

This is needed for supporting the debugger feature, because we
can't allow the CP to preempt queues which are currently being debugged.

The number of queues that can be designated as static is limited by the
number of HQDs (Hardware Queue Descriptors).
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds four new IOCTLs to amdkfd. These IOCTLs expose a H/W
debugger functionality to the userspace.

The IOCTLs are:

- AMDKFD_IOC_DBG_REGISTER:

The purpose of this IOCTL is to notify amdkfd that a process wants to use
GPU debugging facilities on itself only.
It is expected that this IOCTL would be called before any other H/W
debugger requests are sent to amdkfd and for each GPU where the H/W
debugging needs to be enabled. The use of this IOCTL ensures that only
one instance of a debugger is active in the system.

- AMDKFD_IOC_DBG_UNREGISTER:

This IOCTL detaches the debugger/debugged process from the H/W
Debug which was established by the AMDKFD_IOC_DBG_REGISTER IOCTL.

- AMDKFD_IOC_DBG_ADDRESS_WATCH:

This IOCTL allows to set different watchpoints with various conditions as
indicated by the IOCTL's arguments. The available number of watchpoints
is retrieved from topology. This operation is confined to the current
debugged process, which was registered through AMDKFD_IOC_DBG_REGISTER.

- AMDKFD_IOC_DBG_WAVE_CONTROL:

This IOCTL allows to control a wavefront as indicated by the IOCTL's
arguments. For example, you can halt/resume or kill either a
single wavefront or a set of wavefronts. This operation is confined to
the current debugged process, which was registered through
AMDKFD_IOC_DBG_REGISTER.

Because the arguments for the address watch IOCTL and wave control IOCTL
are dynamic, meaning that they could vary in size, the userspace passes a
pointer to a structure (in userspace) that contains the value of the
arguments. The kernel driver is responsible to parse this structure and
validate its contents.

v2: change void* to uint64_t inside ioctl arguments
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Use the generic mechanism to declare a bitmap instead of unsigned long.

It seems that "struct kfd_process.allocated_queue_bitmap" is unused.
Maybe it could be deleted instead.
Signed-off-by: NJoe Perches <joe@perches.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NAndrew Lewycky <Andrew.Lewycky@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds a new kernel module parameter to amdkfd,
called send_sigterm.

This parameter specifies whether amdkfd should send the
SIGTERM signal to an HSA process, when the following conditions
occur:

1. The GPU triggers an exception regarding a kernel that was
   issued by this process.

2. The HSA process isn't waiting on an event that handles
   this exception.

The default behavior is not to send a SIGTERM and suffice
with a dmesg error print.
Reviewed-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NAlexey Skidanov <alexey.skidanov@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds Peripheral Page Request (PPR) failure processing
and reporting.

Bad address or pointer to a system memory block with inappropriate
read/write permission cause such PPR failure during a user queue
processing. PPR request handling is done by IOMMU driver notifying
AMDKFD module on PPR failure.

The process triggering a PPR failure will be notified by
appropriate event or SIGTERM signal will be sent to it.

v3:
- Change all bool fields in struct kfd_memory_exception_failure to
  uint32_t
Signed-off-by: NAlexey Skidanov <alexey.skidanov@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds the events module (kfd_events.c) and the interrupt
handle module for Kaveri (cik_event_interrupt.c).

The patch updates the interrupt_is_wanted(), so that it now calls the
interrupt isr function specific for the device that received the
interrupt. That function(implemented in cik_event_interrupt.c)
returns whether this interrupt is of interest to us or not.

The patch also updates the interrupt_wq(), so that it now calls the
device's specific wq function, which checks the interrupt source
and tries to signal relevant events.

v2:

Increase limit of signal events to 4096 per process
Remove bitfields from struct cik_ih_ring_entry
Rename radeon_kfd_event_mmap to kfd_event_mmap
Add debug prints to allocate_free_slot and allocate_signal_page
Make allocate_event_notification_slot return a correct value
Add warning prints to create_signal_event
Remove error print from IOCTL path
Reformatted debug prints in kfd_event_mmap
Map correct size (as received from mmap) in kfd_event_mmap

v3:

Reduce limit of signal events back to 256 per process
Fix allocation of kernel memory for signal events
Signed-off-by: NAndrew Lewycky <Andrew.Lewycky@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

- AMDKFD_IOC_CREATE_EVENT:
	Creates a new event of a specified type

- AMDKFD_IOC_DESTROY_EVENT:
	Destroys an existing event

- AMDKFD_IOC_SET_EVENT:
	Signal an existing event

- AMDKFD_IOC_RESET_EVENT:
	Reset an existing event

- AMDKFD_IOC_WAIT_EVENTS:
	Wait on event(s) until they are signaled

v2:

- Move the limit of the signal events to kfd_ioctl.h so it
  can be used by userspace

v3:
- Change all bool fields in struct kfd_memory_exception_failure
to uint32_t
Signed-off-by: NAndrew Lewycky <Andrew.Lewycky@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>