Since the assembly code is inside "#if 0", it is ineffective. Despite that,
during debugging, we need to change the assembly code, extract it into
a separate file and compile the new file into hex values using sp3.
That process also requires us to remove "#if 0" and modify lines starting
with "#", so that sp3 can successfully compile the new file.

With this change, all the above chore is no longer needed, and
cwsr_trap_handler_gfx*.asm can be directly used by sp3 to generate its
hex values.
Signed-off-by: NYong Zhao <yong.zhao@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NPhilip Yang <Philip.Yang@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

MTYPE_NC_NV (0) marks scalar/vector L1 cache lines as non-volatile.
Cache lines loaded through these apertures are intended to be
invalidated before (and sometimes during) a dispatch. The non-volatile
qualifier prevents these cache lines from being distinguished from
those loaded through the private aperture.

Use MTYPE_NC (1) instead on both Gfx7 and Gfx8. This allows the
compiler to use the BUFFER_WBINVL1_VOL instruction and is a precursor
to automatic per-dispatch scalar/vector L1 volatile invalidation.
Signed-off-by: NJay Cornwall <Jay.Cornwall@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Synchronization between context-saving wavefronts is achieved by
sending a SAVEWAVE message to the SPI and then spin-waiting for a
response. These spin-waiting wavefronts may inhibit the progress
of other wavefronts in the context save handler, leading to the
synchronization condition never being achieved.

Before spin-waiting reduce the priority of each wavefront to
guarantee foward progress in the others.
Signed-off-by: NJay Cornwall <Jay.Cornwall@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NOak Zeng <Oak.Zeng@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

args->n_devices is a u32 that comes from the user.  The multiplication
could overflow on 32 bit systems possibly leading to privilege
escalation.

Fixes: 5ec7e028 ("drm/amdkfd: Add ioctls for GPUVM memory management")
Signed-off-by: Dan Carpenter dan.carpenter@oracle.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

* Report 64-bit doorbells as HSA_CAP_DOORBELL_TYPE_2_0 in topology
* Report cache information in topology (duplicates GFXv8 info for now)
* Add device info for Vega10 support in KFD

Raven is not enabled at this time as it needs additional changes in
DQM to work with a single SDMA engine.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Report failure to enable atomics only on GPUs that require them.
This allows GPUs that don't require atomics to function, but can
benefit if they are available. This is the case for Vega10, which
doesn't use atomics for basic functioning of the MEC, AQL and HWS
microcode. So it can work without atomics. But shader programs can
still use atomic instructions on systems that support PCIe atomics.
Signed-off-by: Nwelu <Wei.Lu2@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NShaoyun Liu <Shaoyun.Liu@amd.com>
Signed-off-by: NJay Cornwall <Jay.Cornwall@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This condition was missed in a previous commit with the same title.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

v2: Removed redundant 0x before %p.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

kq->queue->properties.write_ptr is a GPU address which can'd be
derefenced in the kernel. Use kq->wptr_kernel instead, which is the
kernel CPU address of the same buffer.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Missed a spot in previous cleanup commit:
Remove gotos that do not feature any common cleanup, and use gotos
instead of repeating cleanup commands.

According to kernel.org: "The goto statement comes in handy when a
function exits from multiple locations and some common work such as
cleanup has to be done. If there is no cleanup needed then just return
directly."
Signed-off-by: NKent Russell <kent.russell@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NShaoyun Liu <Shaoyun.Liu@amd.com>
Signed-off-by: NOak Zeng <Oak.Zeng@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NJohn Bridgman <john.bridgman@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NJohn Bridgman <john.bridgman@amd.com>
Signed-off-by: NJay Cornwall <Jay.Cornwall@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NShaoyun Liu <Shaoyun.Liu@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This is in preparation for GFXv9 (Vega10) which uses incompatible PM4
packet formats from previous ASIC generations.
Signed-off-by: NShaoyun Liu <Shaoyun.Liu@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Allocate doorbells according to the doorbell routing information on
SOC15 ASICs (Vega10 and later). On older ASICs we continue to use the
queue_id as the doorbell ID to maintain compatibility with the Thunk.
Signed-off-by: NShaoyun Liu <Shaoyun.Liu@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Use bit-rotate for better clarity and remove _MASK from the #defines as
these represent mmap types.

Centralize all the parsing of the mmap offset in kfd_mmap and add device
parameter to doorbell and reserved_mem map functions.

Encode gpu_id into upper bits of vm_pgoff. This frees up the lower bits
for encoding the the doorbell ID on Vega10.
Signed-off-by: NHarish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This prepares for GFXv9 (Vega10), which has 64-bit doorbells.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This is needed for Vega10 and later ASICs to let KFD know which
doorbells can be used for SDMA and CP queues respectively.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NJohn Bridgman <john.bridgman@amd.com>
Signed-off-by: NShaoyun Liu <Shaoyun.Liu@amd.com>
Signed-off-by: NJay Cornwall <Jay.Cornwall@amd.com>
Signed-off-by: NYong Zhao <yong.zhao@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NShaoyun Liu <Shaoyun.Liu@amd.com>
Signed-off-by: NJay Cornwall <Jay.Cornwall@amd.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This matches what the HWS firmware expects on GFXv9 chips.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This adds support for allocating, mapping, unmapping and freeing
userptr BOs, and for handling MMU notifiers.

v2: updated a comment
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

These interfaces allow KGD to stop and resume all GPU user mode queue
access to a process address space. This is needed for handling MMU
notifiers of userptrs mapped for GPU access in KFD VMs.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

When an MMU notifier runs in memory reclaim context, it can deadlock
trying to take locks that are already held in the thread causing the
memory reclaim. The solution is to avoid memory reclaim while holding
locks that are taken in MMU notifiers by using GFP_NOIO.

This commit fixes memory allocations done while holding the dqm->lock
which is needed in the MMU notifier (dqm->ops.evict_process_queues).
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

When an MMU notifier runs in memory reclaim context, it can deadlock
trying to take locks that are already held in the thread causing the
memory reclaim. The solution is to avoid memory reclaim while holding
locks that are taken in MMU notifiers.

This commit fixes kmalloc while holding rmn->lock by moving the call
outside the lock. The GFX MMU notifier also locks reservation objects.
I have no good solution for avoiding reclaim while holding reservation
objects. The HSA MMU notifier will not lock any reservation objects.

v2: Moved allocation outside lock instead of using GFP_NOIO
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NOded Gabbay <oded.gabbay@gmail.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This commit allows amdgpu_ttm_tt_get_user_pages to work in a worker
thread rather than regular process context. This will be used when
KFD userptr BOs are restored after an MMU-notifier eviction.

v2: Manage task reference with get_task_struct/put_task_struct
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NOded Gabbay <oded.gabbay@gmail.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

This commit adds the notion of MMU notifier types GFX and HSA. GFX
continues to work like MMU notifiers did before. HSA adds support for
KFD userptr BOs. The implementation of KFD userptr eviction is a stub
for now.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: NOded Gabbay <oded.gabbay@gmail.com>

Pull drm fixes from Dave Airlie:
 "Exynos, i915, vc4, amdgpu fixes.

  i915:
   - an oops fix
   - two race fixes
   - some gvt fixes

  amdgpu:
   - dark screen fix
   - clk/voltage fix
   - vega12 smu fix

  vc4:
   - memory leak fix

  exynos just drops some code"

* tag 'drm-fixes-for-v4.17-rc2' of git://people.freedesktop.org/~airlied/linux: (23 commits)
  drm/amd/powerplay: header file interface to SMU update
  drm/amd/pp: Fix bug voltage can't be OD separately on VI
  drm/amd/display: Don't program bypass on linear regamma LUT
  drm/i915: Fix LSPCON TMDS output buffer enabling from low-power state
  drm/i915/audio: Fix audio detection issue on GLK
  drm/i915: Call i915_perf_fini() on init_hw error unwind
  drm/i915/bios: filter out invalid DDC pins from VBT child devices
  drm/i915/pmu: Inspect runtime PM state more carefully while estimating RC6
  drm/i915: Do no use kfree() to free a kmem_cache_alloc() return value
  drm/exynos: exynos_drm_fb -> drm_framebuffer
  drm/exynos: Move dma_addr out of exynos_drm_fb
  drm/exynos: Move GEM BOs to drm_framebuffer
  drm: Fix HDCP downstream dev count read
  drm/vc4: Fix memory leak during BO teardown
  drm/i915/execlists: Clear user-active flag on preemption completion
  drm/i915/gvt: Add drm_format_mod update
  drm/i915/gvt: Disable primary/sprite/cursor plane at virtual display initialization
  drm/i915/gvt: Delete redundant error message in fb_decode.c
  drm/i915/gvt: Cancel dma map when resetting ggtt entries
  drm/i915/gvt: Missed to cancel dma map for ggtt entries
  ...

- Fix a dark screen issue in DC
- Fix clk/voltage dependency tracking for wattman
- Update SMU interface for vega12

* 'drm-next-4.17' of git://people.freedesktop.org/~agd5f/linux:
  drm/amd/powerplay: header file interface to SMU update
  drm/amd/pp: Fix bug voltage can't be OD separately on VI
  drm/amd/display: Don't program bypass on linear regamma LUT

Merge tag 'exynos-drm-fixes-for-v4.17-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/daeinki/drm-exynos into drm-next

Remove Exynos specific framebuffer structure and
relevant functions.
- it removes exynos_drm_fb structure which is a wrapper of
  drm_framebuffer and unnecessary two exynos specific callback
  functions, exynos_drm_destory() and exynos_drm_fb_create_handle()
  because we can reuse existing drm common callback ones instead.

* tag 'exynos-drm-fixes-for-v4.17-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/daeinki/drm-exynos:
  drm/exynos: exynos_drm_fb -> drm_framebuffer
  drm/exynos: Move dma_addr out of exynos_drm_fb
  drm/exynos: Move GEM BOs to drm_framebuffer
  drm/amdkfd: Deallocate SDMA queues correctly
  drm/amdkfd: Fix scratch memory with HWS enabled

Merge tag 'drm-intel-next-fixes-2018-04-19' of git://anongit.freedesktop.org/drm/drm-intel into drm-next

- Fix for FDO #105549: Avoid OOPS on bad VBT (Jani)
- Fix rare pre-emption race (Chris)
- Fix RC6 race against PM transitions (Tvrtko)

* tag 'drm-intel-next-fixes-2018-04-19' of git://anongit.freedesktop.org/drm/drm-intel:
  drm/i915/audio: Fix audio detection issue on GLK
  drm/i915: Call i915_perf_fini() on init_hw error unwind
  drm/i915/bios: filter out invalid DDC pins from VBT child devices
  drm/i915/pmu: Inspect runtime PM state more carefully while estimating RC6
  drm/i915: Do no use kfree() to free a kmem_cache_alloc() return value
  drm/i915/execlists: Clear user-active flag on preemption completion
  drm/i915/gvt: Add drm_format_mod update
  drm/i915/gvt: Disable primary/sprite/cursor plane at virtual display initialization
  drm/i915/gvt: Delete redundant error message in fb_decode.c
  drm/i915/gvt: Cancel dma map when resetting ggtt entries
  drm/i915/gvt: Missed to cancel dma map for ggtt entries
  drm/i915/gvt: Make MI_USER_INTERRUPT nop in cmd parser
  drm/i915/gvt: Mark expected switch fall-through in handle_g2v_notification
  drm/i915/gvt: throw error on unhandled vfio ioctls

drm-misc-fixes:

stable: vc4: Fix memory leak during BO teardown (Daniel)
dp: Add i2c retry for LSPCON adapters (Imre)
hdcp: Fix device count mask (Ramalingam)

Cc: Daniel J Blueman <daniel@quora.org
Cc: Imre Deak <imre.deak@intel.com>
Cc: Ramalingam C <ramalingam.c@intel.com>

* tag 'drm-misc-fixes-2018-04-18-1' of git://anongit.freedesktop.org/drm/drm-misc:
  drm/i915: Fix LSPCON TMDS output buffer enabling from low-power state
  drm: Fix HDCP downstream dev count read
  drm/vc4: Fix memory leak during BO teardown