This patch adds four new IOCTLs to amdkfd. These IOCTLs expose a H/W
debugger functionality to the userspace.

The IOCTLs are:

- AMDKFD_IOC_DBG_REGISTER:

The purpose of this IOCTL is to notify amdkfd that a process wants to use
GPU debugging facilities on itself only.
It is expected that this IOCTL would be called before any other H/W
debugger requests are sent to amdkfd and for each GPU where the H/W
debugging needs to be enabled. The use of this IOCTL ensures that only
one instance of a debugger is active in the system.

- AMDKFD_IOC_DBG_UNREGISTER:

This IOCTL detaches the debugger/debugged process from the H/W
Debug which was established by the AMDKFD_IOC_DBG_REGISTER IOCTL.

- AMDKFD_IOC_DBG_ADDRESS_WATCH:

This IOCTL allows to set different watchpoints with various conditions as
indicated by the IOCTL's arguments. The available number of watchpoints
is retrieved from topology. This operation is confined to the current
debugged process, which was registered through AMDKFD_IOC_DBG_REGISTER.

- AMDKFD_IOC_DBG_WAVE_CONTROL:

This IOCTL allows to control a wavefront as indicated by the IOCTL's
arguments. For example, you can halt/resume or kill either a
single wavefront or a set of wavefronts. This operation is confined to
the current debugged process, which was registered through
AMDKFD_IOC_DBG_REGISTER.

Because the arguments for the address watch IOCTL and wave control IOCTL
are dynamic, meaning that they could vary in size, the userspace passes a
pointer to a structure (in userspace) that contains the value of the
arguments. The kernel driver is responsible to parse this structure and
validate its contents.

v2: change void* to uint64_t inside ioctl arguments
Signed-off-by: NYair Shachar <yair.shachar@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NAndrew Lewycky <Andrew.Lewycky@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This patch adds the events module (kfd_events.c) and the interrupt
handle module for Kaveri (cik_event_interrupt.c).

The patch updates the interrupt_is_wanted(), so that it now calls the
interrupt isr function specific for the device that received the
interrupt. That function(implemented in cik_event_interrupt.c)
returns whether this interrupt is of interest to us or not.

The patch also updates the interrupt_wq(), so that it now calls the
device's specific wq function, which checks the interrupt source
and tries to signal relevant events.

v2:

Increase limit of signal events to 4096 per process
Remove bitfields from struct cik_ih_ring_entry
Rename radeon_kfd_event_mmap to kfd_event_mmap
Add debug prints to allocate_free_slot and allocate_signal_page
Make allocate_event_notification_slot return a correct value
Add warning prints to create_signal_event
Remove error print from IOCTL path
Reformatted debug prints in kfd_event_mmap
Map correct size (as received from mmap) in kfd_event_mmap

v3:

Reduce limit of signal events back to 256 per process
Fix allocation of kernel memory for signal events
Signed-off-by: NAndrew Lewycky <Andrew.Lewycky@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

- AMDKFD_IOC_CREATE_EVENT:
	Creates a new event of a specified type

- AMDKFD_IOC_DESTROY_EVENT:
	Destroys an existing event

- AMDKFD_IOC_SET_EVENT:
	Signal an existing event

- AMDKFD_IOC_RESET_EVENT:
	Reset an existing event

- AMDKFD_IOC_WAIT_EVENTS:
	Wait on event(s) until they are signaled

v2:

- Move the limit of the signal events to kfd_ioctl.h so it
  can be used by userspace

v3:
- Change all bool fields in struct kfd_memory_exception_failure
to uint32_t
Signed-off-by: NAndrew Lewycky <Andrew.Lewycky@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

The current code can only support one kgd instance. We have to
support multiple kgd instances in one system. i.e two amdgpu or two
radeon or one amdgpu + one radeon or more than two kgd instances.
Signed-off-by: NXihan Zhang <xihan.zhang@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Convert the timestamping in the amdkfd driver to use a timespec64 and 64bit
time accessors.

Although the existing code is completely safe beyond y2038 because it deals
with monotonic time, this patch is still needed in order to kill off all uses
of struct timespec.
Signed-off-by: NJohn Stultz <john.stultz@linaro.org>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch passes the correct queue type to pqm_create_queue() instead of a
fixed KFD_QUEUE_TYPE_COMPUTE type.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch adds a check to the create queue ioctl path, which identifies SDMA
queue type that is sent by userspace.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

This patch changes kfd_ioctl() to be very similar to drm_ioctl().

The patch defines an array of amdkfd_ioctls, which maps IOCTL definition to the
ioctl function.

The kfd_ioctl() uses that mapping to call the appropriate ioctl function,
through a function pointer.

This patch also declares a new typedef for the ioctl function pointer.

v2: Renamed KFD_COMMAND_(START|END) to AMDKFD_...
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>

This patch reformats the ioctl definitions in kfd_ioctl.h to be similar to the
drm ioctls definition style.

v2: Renamed KFD_COMMAND_(START|END) to AMDKFD_...
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>

This patch moves the copy_to_user() and copy_from_user() calls from the
different ioctl functions in amdkfd to the general kfd_ioctl() function, as
this is a common code for all ioctls.

This was done according to example taken from drm_ioctl.c
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>

This patch checks if the process that opens the /dev/kfd device is 32-bit
process. If so, it returns -EPERM and prints a warning message in dmesg.

This is done to prevent 32-bit user processes from using amdkfd, and hence, HSA
features.

AMD's HSA userspace stack will also support only 64-bit processes on Linux.
Reviewed-by: NAlexey Skidanov <alexey.skidanov@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch does some re-org on the device_queue_manager structure. It takes out
all the function pointers from the structure and puts them in a new structure,
called device_queue_manager_ops. Then, it puts an instance of that structure
inside device_queue_manager.

This re-org is done to prepare the DQM module to support more than one AMD APU
(Kaveri).
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

There is a typo here so the errors from kfd_bind_process_to_device()
are not detected.
Reviewed-by: NOded Gabbay <oded.gabbay@amd.com>
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Since the user space may call open() more that once from the same process,
the aperture initialization should be moved from kfd_open()
Signed-off-by: NAlexey Skidanov <Alexey.Skidanov@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch adds new fields to the queue_properties structure. The new fields
are relevant only for queues running on AMD GPU VI architecture.

The eop_ring_buffer_address and eop_ring_buffer_size describe an
end-of-pipe queue which is assigned to the MQD. In CI, the EOP queue was per
pipeline and in VI it is per queue.

The ctx_save_restore_area_address and ctx_save_restore_area_size describe a
memory area that is designated to allow the CP to do context save/restore in
mid-wave state.

This patch also modifies the set_queue_properties_from_user() (called from
kfd_ioctl_create_queue()) to check and copy those new parameters.
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

v3: Fixed debug messages
Signed-off-by: NAlexey Skidanov <Alexey.Skidanov@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Signed-off-by: NEvgeny Pinchuk <evgeny.pinchuk@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Signed-off-by: NAndrew Lewycky <Andrew.Lewycky@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

v3: Removed the use of internal typedefs, fixed debug prints, added checks
    for parameters and moved to using doorbell address from user

v4: Extracted some of the code in the create queue ioctl to a different
    function that may be also called from other ioctls in the future.
    Also fixed the check of the ring size argument.

v5:

Add support for AQL queues creation to enable working with open-source HSA
runtime
Signed-off-by: NBen Goz <ben.goz@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch adds the process module and three helper modules:

- kfd_process, which handles process which open /dev/kfd

- kfd_doorbell, which provides helper functions for doorbell allocation,
  release and mapping to userspace

- kfd_pasid, which provides helper functions for pasid allocation and release

- kfd_aperture, which provides helper functions for managing the LDS, Local GPU
  memory and Scratch memory apertures of the process

This patch only contains the basic kfd_process module, which doesn't contain
the reference to the queue scheduler. This was done to allow easier code review.

Also, this patch doesn't contain the calls to the IOMMU driver for binding the
pasid to the device. Again, this was done to allow easier code review

The kfd_process object is created when a process opens /dev/kfd and is closed
when the mm_struct of that process is teared-down.

v3:

Removed kfd_vidmem.c file
Replaced direct mmput call to mmu_notifier release
Removed typedefs
Moved bool field to end of the structure
Added new kernel params for gart usage limitation
Added initialization of sa manager
Fixed debug messages
Remove support for LDS in 32 bit
Changed code to support mmap of doorbell pages from userspace
Added documentation for apertures

v4: Replaced RCU by SRCU for kfd_process list management

v5:

Move amdkfd from drm/radeon/ to drm/amd/
Rename kfd_aperture.c to kfd_flat_memory.c
Protect against multiple init calls
MQD size is H/W dependent so moved it to device info structure
Rename kfd_mem_obj structure's members
Use delayed function for process tear-down
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch adds the amdkfd skeleton driver. The driver does nothing except
define a /dev/kfd device.

It returns -ENODEV on all amdkfd IOCTLs.

v3: Move bool field to the end of structure, removed the pmc ioctls and added
a meaningful error message for ioctl error.

v5:

Create a new folder drm/amd and move amdkfd from drm/radeon/ to drm/amd/
Remove scheduler_class from kfd_priv.h as it was never used
Add skeleton implementation of the Get Version IOCTL

v6:
Update module version to the correct number and remove the "default m" from the
Kconfig file
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>