This patch adds the interface between the radeon driver and the amdkfd driver.
The interface implementation is contained in radeon_kfd.c and radeon_kfd.h.

The interface itself is represented by a pointer to struct
kfd_dev. The pointer is located inside radeon_device structure.

All the register accesses that amdkfd need are done using this interface. This
allows us to avoid direct register accesses in amdkfd proper,  while also
avoiding locking between amdkfd and radeon.

The single exception is the doorbells that are used in both of the drivers.
However, because they are located in separate pci bar pages, the danger of
sharing registers between the drivers is minimal.

Having said that, we are planning to move the doorbells as well to radeon.

v3:

Add interface for sa manager init and fini. The init function will allocate a
buffer on system memory and pin it to the GART address space via the radeon sa
manager.

All mappings of buffers to GART address space are done via the radeon sa
manager. The interface of allocate memory will use the radeon sa manager to sub
allocate from the single buffer that was allocated during the init function.

Change lower_32/upper_32 calls to use linux macros

Add documentation for the interface

v4:

Change ptr field type in kgd_mem from uint32_t* to void* to match to type that
is returned by radeon_sa_bo_cpu_addr

v5:

Change format of mqd structure to work with latest KV firmware
Add support for AQL queues creation to enable working with open-source HSA
runtime.
Move generic kfd-->kgd interface and other generic kgd definitions to a generic
header file that will be used by AMD's radeon and amdgpu drivers
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Implementing a lock for selecting and accessing shader engines and arrays.
This lock will make sure that radeon and amdkfd are not colliding when
accessing shader engines and arrays with GRBM_GFX_INDEX register.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

radeon and amdkfd share the doorbell aperture.
radeon sets it up, takes the doorbells required for its own rings
and reports the setup to amdkfd.
radeon reserved doorbells are at the start of the doorbell aperture.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

amdkfd should set interrupts for pipes 1-7.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

To support HSA on KV, we need to limit the number of vmids and pipes
that are available for radeon's use with KV.

This patch reserves VMIDs 8-15 for amdkfd (so radeon can only use VMIDs
0-7) and also makes radeon thinks that KV has only a single MEC with a single
pipe in it

v3: Use define for static vmid allocation in radeon
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This patch fixes a bug in the accounting of the device_state.
In the current code, the device_state was put (decremented) too many times,
which sometimes lead to the driver getting stuck permanently in
put_device_state_wait(). That happen because the device_state->count would go
below zero, which is never supposed to happen.

The root cause is that the device_state was decremented in put_pasid_state()
and put_pasid_state_wait() but also in all the functions that call those
functions. Therefore, the device_state was decremented twice in each of these
code paths.

The fix is to decouple the device_state accounting from the pasid_state
accounting - remove the call to put_device_state() from the
put_pasid_state() and the put_pasid_state_wait())
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Make use of the new invalidate_range mmu_notifier call-back and remove the
old logic of assigning an empty page-table between invalidate_range_start
and invalidate_range_end.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>
Tested-by: NOded Gabbay <oded.gabbay@amd.com>
Reviewed-by: NAndrea Arcangeli <aarcange@redhat.com>
Reviewed-by: NJérôme Glisse <jglisse@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Rik van Riel <riel@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Jay Cornwall <Jay.Cornwall@amd.com>
Cc: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: David Woodhouse <dwmw2@infradead.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Now that the mmu_notifier_invalidate_range() calls are in place, add the
callback to allow subsystems to register against it.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>
Reviewed-by: NAndrea Arcangeli <aarcange@redhat.com>
Reviewed-by: NJérôme Glisse <jglisse@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Rik van Riel <riel@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Jay Cornwall <Jay.Cornwall@amd.com>
Cc: Oded Gabbay <Oded.Gabbay@amd.com>
Cc: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: David Woodhouse <dwmw2@infradead.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Add calls to the new mmu_notifier_invalidate_range() function to all
places in the VMM that need it.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>
Reviewed-by: NAndrea Arcangeli <aarcange@redhat.com>
Reviewed-by: NJérôme Glisse <jglisse@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Rik van Riel <riel@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Jay Cornwall <Jay.Cornwall@amd.com>
Cc: Oded Gabbay <Oded.Gabbay@amd.com>
Cc: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: David Woodhouse <dwmw2@infradead.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

This notifier closes an important gap in the current mmu_notifier
implementation, the existing callbacks are called too early or too late to
reliably manage a non-CPU TLB.  Specifically, invalidate_range_start() is
called when all pages are still mapped and invalidate_range_end() when all
pages are unmapped and potentially freed.

This is fine when the users of the mmu_notifiers manage their own SoftTLB,
like KVM does.  When the TLB is managed in software it is easy to wipe out
entries for a given range and prevent new entries to be established until
invalidate_range_end is called.

But when the user of mmu_notifiers has to manage a hardware TLB it can
still wipe out TLB entries in invalidate_range_start, but it can't make
sure that no new TLB entries in the given range are established between
invalidate_range_start and invalidate_range_end.

To avoid silent data corruption the entries in the non-CPU TLB need to be
flushed when the pages are unmapped (at this point in time no _new_ TLB
entries can be established in the non-CPU TLB) but not yet freed (as the
non-CPU TLB may still have _existing_ entries pointing to the pages about
to be freed).

To fix this problem we need to catch the moment when the Linux VMM flushes
remote TLBs (as a non-CPU TLB is not very CPU TLB), as this is the point
in time when the pages are unmapped but _not_ yet freed.

The mmu_notifier_invalidate_range() function aims to catch that moment.

IOMMU code will be one user of the notifier-callback.  Currently this is
only the AMD IOMMUv2 driver, but its code is about to be more generalized
and converted to a generic IOMMU-API extension to fit the needs of similar
functionality in other IOMMUs as well.

The current attempt in the AMD IOMMUv2 driver to work around the
invalidate_range_start/end() shortcoming is to assign an empty page table
to the non-CPU TLB between any invalidata_range_start/end calls.  With the
empty page-table assigned, every page-table walk to re-fill the non-CPU
TLB will cause a page-fault reported to the IOMMU driver via an interrupt,
possibly causing interrupt storms.

The page-fault handler in the AMD IOMMUv2 driver doesn't handle the fault
if an invalidate_range_start/end pair is active, it just reports back
SUCCESS to the device and let it refault the page.  But existing hardware
(newer Radeon GPUs) that makes use of this feature don't re-fault
indefinitly, after a certain number of faults for the same address the
device enters a failure state and needs to be resetted.

To avoid the GPUs entering a failure state we need to get rid of the
empty-page-table workaround and use the mmu_notifier_invalidate_range()
function introduced with this patch.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>
Reviewed-by: NAndrea Arcangeli <aarcange@redhat.com>
Reviewed-by: NJérôme Glisse <jglisse@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Rik van Riel <riel@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Jay Cornwall <Jay.Cornwall@amd.com>
Cc: Oded Gabbay <Oded.Gabbay@amd.com>
Cc: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: David Woodhouse <dwmw2@infradead.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Radeon patches for 3.19.  Christian has a number of GPUVM improvements
slated as well, but I'd like to wait until he gets back to work next week
to pull those in. Highlights of this pull:
- ttm performance improvements
- CI dpm fixes

* 'drm-next-3.19' of git://people.freedesktop.org/~agd5f/linux: (26 commits)
  drm/radeon/si/ci: make u8 static arrays constant
  drm/radeon: set power control in ci dpm enable
  drm/radeon: powertune fixes for hawaii
  drm/radeon: fix dpm mc init for certain hawaii boards
  drm/radeon: set bootup pcie level to max for ci dpm
  drm/radeon: fix default dpm state setup
  drm/radeon: workaround a hw bug in bonaire pcie dpm
  drm/radeon: fix mclk vddc configuration for cards for hawaii
  drm/radeon: fix sclk DS enablement
  drm/radeon: fix activity settings for sclk and mclk for CI
  drm/radeon: improve mclk param calcuations for ci dpm
  drm/radeon: fix dram timing for certain hawaii boards
  drm/radeon: switch force state commands for CI
  drm/radeon: fix for memory training on bonaire 0x6649
  drm/radeon/ci: handle gpio controlled dpm features properly
  drm/radeon: store the gpio shift as well
  drm/radeon: export radeon_atombios_lookup_gpio
  drm/radeon: fix typo in CI dpm disable
  drm/radeon: rework CI dpm thermal setup
  drm/radeon: rework SI dpm thermal setup
  ...

These two arrays don't change, just make them constant,
reduces data segment by a few bytes.
Signed-off-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Necessary for poper operation.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

- bapm is not available on hawaii
- update pt defaults
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Needs special overrides for certain vram configurations.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Avoids problems when re-loading the driver.  Does not
affect power saving when dpm is enabled.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Only enable the first levels for mclk and sclk.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Some boards get stuck in pcie x1 otherwise.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Need to use vddc0 for vdcc1 for certain hawaii configurations.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Only enable it for levels 0 and 1.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Only need to be enabled on the first level.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Properly take into account the post divider.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Certain memory configurations need a fix.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Use the preferred SMC commands for forcing state on CI.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Workaround for memory link training on certain variants
of 0x6649.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Certain feature enablement depends on entries in the atom
gpio pin table.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We need this in the dpm code.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We need it for dpm.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Need to disable DS, not enable it when disabling dpm.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

In preparation for fan control.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

In preparation for fan control.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Required for fan control support.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

DRM_MM_SEARCH_BEST gets the smallest hole which can fit the BO. That seems
against the idea of TTM_PL_FLAG_TOPDOWN:

* The smallest hole may be in the overall bottom of the area
* If the hole isn't much larger than the BO, it doesn't make much
  difference whether the BO is placed at the bottom or at the top of the
  hole
Reviewed-by: NLauri Kasanen <cand@gmx.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

If the BO should be placed at the top of the area, we should start looking
for holes from the top.
Reviewed-by: NLauri Kasanen <cand@gmx.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

I wasn't sure if TTM_PL_FLAG_TOPDOWN works correctly with non-0 lpfn, but
AFAICT it does.
Reviewed-by: NLauri Kasanen <cand@gmx.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This avoids them getting in the way of BOs which might be accessed by
the CPU. They can still go to the CPU accessible part of VRAM though if
there's no space outside of it.
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Motivated by the per-plane locking I've gone through all the get*
ioctls and reduced the locking to the bare minimum required.

v2: Rebase and make it compile ...

v3: Review from Sean:
- Simplify return handling in getplane_res.
- Add a comment to getplane_res that the plane list is invariant and
  can be walked locklessly.

v4: Actually git add.

Cc: Sean Paul <seanpaul@chromium.org>
Reviewed-by: NSean Paul <seanpaul@chromium.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@intel.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Turned out to be much simpler on top of my latest atomic stuff than
what I've feared. Some details:

- Drop the modeset_lock_all snakeoil in drm_plane_init. Same
  justification as for the equivalent change in drm_crtc_init done in

	commit d0fa1af4
	Author: Daniel Vetter <daniel.vetter@ffwll.ch>
	Date:   Mon Sep 8 09:02:49 2014 +0200

	    drm: Drop modeset locking from crtc init function

  Without these the drm_modeset_lock_init would fall over the exact
  same way.

- Since the atomic core code wraps the locking switching it to
  per-plane locks was a one-line change.

- For the legacy ioctls add a plane argument to the locking helper so
  that we can grab the right plane lock (cursor or primary). Since the
  universal cursor plane might not be there, or someone really crazy
  might forgoe the primary plane even accept NULL.

- Add some locking WARN_ON to the atomic helpers for good paranoid
  measure and to check that it all works out.

Tested on my exynos atomic hackfest with full lockdep checks and ww
backoff injection.

v2: I've forgotten about the load-detect code in i915.

v3: Thierry reported that in latest 3.18-rc vmwgfx doesn't compile any
more due to

commit 21e88620
Author: Rob Clark <robdclark@gmail.com>
Date:   Thu Oct 30 13:39:04 2014 -0400

    drm/vmwgfx: fix lock breakage

Rebased and fix this up.

Cc: Thierry Reding <thierry.reding@gmail.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@intel.com>
Reviewed-by: NSean Paul <seanpaul@chromium.org>
Signed-off-by: NDave Airlie <airlied@redhat.com>

v1: original
v2: danvet's kerneldoc nitpicks
Signed-off-by: NRob Clark <robdclark@gmail.com>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NDave Airlie <airlied@redhat.com>