This notifier closes an important gap in the current mmu_notifier
implementation, the existing callbacks are called too early or too late to
reliably manage a non-CPU TLB.  Specifically, invalidate_range_start() is
called when all pages are still mapped and invalidate_range_end() when all
pages are unmapped and potentially freed.

This is fine when the users of the mmu_notifiers manage their own SoftTLB,
like KVM does.  When the TLB is managed in software it is easy to wipe out
entries for a given range and prevent new entries to be established until
invalidate_range_end is called.

But when the user of mmu_notifiers has to manage a hardware TLB it can
still wipe out TLB entries in invalidate_range_start, but it can't make
sure that no new TLB entries in the given range are established between
invalidate_range_start and invalidate_range_end.

To avoid silent data corruption the entries in the non-CPU TLB need to be
flushed when the pages are unmapped (at this point in time no _new_ TLB
entries can be established in the non-CPU TLB) but not yet freed (as the
non-CPU TLB may still have _existing_ entries pointing to the pages about
to be freed).

To fix this problem we need to catch the moment when the Linux VMM flushes
remote TLBs (as a non-CPU TLB is not very CPU TLB), as this is the point
in time when the pages are unmapped but _not_ yet freed.

The mmu_notifier_invalidate_range() function aims to catch that moment.

IOMMU code will be one user of the notifier-callback.  Currently this is
only the AMD IOMMUv2 driver, but its code is about to be more generalized
and converted to a generic IOMMU-API extension to fit the needs of similar
functionality in other IOMMUs as well.

The current attempt in the AMD IOMMUv2 driver to work around the
invalidate_range_start/end() shortcoming is to assign an empty page table
to the non-CPU TLB between any invalidata_range_start/end calls.  With the
empty page-table assigned, every page-table walk to re-fill the non-CPU
TLB will cause a page-fault reported to the IOMMU driver via an interrupt,
possibly causing interrupt storms.

The page-fault handler in the AMD IOMMUv2 driver doesn't handle the fault
if an invalidate_range_start/end pair is active, it just reports back
SUCCESS to the device and let it refault the page.  But existing hardware
(newer Radeon GPUs) that makes use of this feature don't re-fault
indefinitly, after a certain number of faults for the same address the
device enters a failure state and needs to be resetted.

To avoid the GPUs entering a failure state we need to get rid of the
empty-page-table workaround and use the mmu_notifier_invalidate_range()
function introduced with this patch.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>
Reviewed-by: NAndrea Arcangeli <aarcange@redhat.com>
Reviewed-by: NJérôme Glisse <jglisse@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Rik van Riel <riel@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Jay Cornwall <Jay.Cornwall@amd.com>
Cc: Oded Gabbay <Oded.Gabbay@amd.com>
Cc: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: David Woodhouse <dwmw2@infradead.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>

Radeon patches for 3.19.  Christian has a number of GPUVM improvements
slated as well, but I'd like to wait until he gets back to work next week
to pull those in. Highlights of this pull:
- ttm performance improvements
- CI dpm fixes

* 'drm-next-3.19' of git://people.freedesktop.org/~agd5f/linux: (26 commits)
  drm/radeon/si/ci: make u8 static arrays constant
  drm/radeon: set power control in ci dpm enable
  drm/radeon: powertune fixes for hawaii
  drm/radeon: fix dpm mc init for certain hawaii boards
  drm/radeon: set bootup pcie level to max for ci dpm
  drm/radeon: fix default dpm state setup
  drm/radeon: workaround a hw bug in bonaire pcie dpm
  drm/radeon: fix mclk vddc configuration for cards for hawaii
  drm/radeon: fix sclk DS enablement
  drm/radeon: fix activity settings for sclk and mclk for CI
  drm/radeon: improve mclk param calcuations for ci dpm
  drm/radeon: fix dram timing for certain hawaii boards
  drm/radeon: switch force state commands for CI
  drm/radeon: fix for memory training on bonaire 0x6649
  drm/radeon/ci: handle gpio controlled dpm features properly
  drm/radeon: store the gpio shift as well
  drm/radeon: export radeon_atombios_lookup_gpio
  drm/radeon: fix typo in CI dpm disable
  drm/radeon: rework CI dpm thermal setup
  drm/radeon: rework SI dpm thermal setup
  ...

These two arrays don't change, just make them constant,
reduces data segment by a few bytes.
Signed-off-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Necessary for poper operation.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

- bapm is not available on hawaii
- update pt defaults
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Needs special overrides for certain vram configurations.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Avoids problems when re-loading the driver.  Does not
affect power saving when dpm is enabled.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Only enable the first levels for mclk and sclk.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Some boards get stuck in pcie x1 otherwise.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Need to use vddc0 for vdcc1 for certain hawaii configurations.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Only enable it for levels 0 and 1.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Only need to be enabled on the first level.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Properly take into account the post divider.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Certain memory configurations need a fix.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Use the preferred SMC commands for forcing state on CI.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Workaround for memory link training on certain variants
of 0x6649.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Certain feature enablement depends on entries in the atom
gpio pin table.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We need this in the dpm code.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We need it for dpm.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Need to disable DS, not enable it when disabling dpm.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

In preparation for fan control.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

In preparation for fan control.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Required for fan control support.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

DRM_MM_SEARCH_BEST gets the smallest hole which can fit the BO. That seems
against the idea of TTM_PL_FLAG_TOPDOWN:

* The smallest hole may be in the overall bottom of the area
* If the hole isn't much larger than the BO, it doesn't make much
  difference whether the BO is placed at the bottom or at the top of the
  hole
Reviewed-by: NLauri Kasanen <cand@gmx.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

If the BO should be placed at the top of the area, we should start looking
for holes from the top.
Reviewed-by: NLauri Kasanen <cand@gmx.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

I wasn't sure if TTM_PL_FLAG_TOPDOWN works correctly with non-0 lpfn, but
AFAICT it does.
Reviewed-by: NLauri Kasanen <cand@gmx.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This avoids them getting in the way of BOs which might be accessed by
the CPU. They can still go to the CPU accessible part of VRAM though if
there's no space outside of it.
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Motivated by the per-plane locking I've gone through all the get*
ioctls and reduced the locking to the bare minimum required.

v2: Rebase and make it compile ...

v3: Review from Sean:
- Simplify return handling in getplane_res.
- Add a comment to getplane_res that the plane list is invariant and
  can be walked locklessly.

v4: Actually git add.

Cc: Sean Paul <seanpaul@chromium.org>
Reviewed-by: NSean Paul <seanpaul@chromium.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@intel.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Turned out to be much simpler on top of my latest atomic stuff than
what I've feared. Some details:

- Drop the modeset_lock_all snakeoil in drm_plane_init. Same
  justification as for the equivalent change in drm_crtc_init done in

	commit d0fa1af4
	Author: Daniel Vetter <daniel.vetter@ffwll.ch>
	Date:   Mon Sep 8 09:02:49 2014 +0200

	    drm: Drop modeset locking from crtc init function

  Without these the drm_modeset_lock_init would fall over the exact
  same way.

- Since the atomic core code wraps the locking switching it to
  per-plane locks was a one-line change.

- For the legacy ioctls add a plane argument to the locking helper so
  that we can grab the right plane lock (cursor or primary). Since the
  universal cursor plane might not be there, or someone really crazy
  might forgoe the primary plane even accept NULL.

- Add some locking WARN_ON to the atomic helpers for good paranoid
  measure and to check that it all works out.

Tested on my exynos atomic hackfest with full lockdep checks and ww
backoff injection.

v2: I've forgotten about the load-detect code in i915.

v3: Thierry reported that in latest 3.18-rc vmwgfx doesn't compile any
more due to

commit 21e88620
Author: Rob Clark <robdclark@gmail.com>
Date:   Thu Oct 30 13:39:04 2014 -0400

    drm/vmwgfx: fix lock breakage

Rebased and fix this up.

Cc: Thierry Reding <thierry.reding@gmail.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@intel.com>
Reviewed-by: NSean Paul <seanpaul@chromium.org>
Signed-off-by: NDave Airlie <airlied@redhat.com>

v1: original
v2: danvet's kerneldoc nitpicks
Signed-off-by: NRob Clark <robdclark@gmail.com>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NDave Airlie <airlied@redhat.com>

backmerge to get vmwgfx locking changes into next as the
conflict with per-plane locking.

These two didn't get documented properly, do so.

Pointed out by Daniel.

v1.1: add missing boilerplate (Daniel)
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Merge tag 'topic/atomic-helpers-2014-11-09' of git://anongit.freedesktop.org/drm-intel into drm-next

So here's my atomic series, finally all debugged&reviewed. Sean Paul has
done a full detailed pass over it all, and a lot of other people have
commented and provided feedback on some parts. Rob Clark also converted
msm over the w/e and seems happy. The only small thing is that Rob wants
to export the wait_for_vblank, which imo makes sense. Since there's other
stuff still to do I think we should apply Rob's patch (once it has grown
appropriate kerneldoc) later on top of this.

This is just the core<->driver interface plus a big pile of helpers. Short
recap of the main ideas:

- There are essentially three helper libraries in this patch set:

  * Transitional helpers to use the new plane callbacks for legacy plane
    updates and in the crtc helper's ->mode_set callback. These helpers are
    only temporarily used to convert drivers to atomic, but they allow a
    nice separation between changing the driver backend and switching to
    the atomic commit logic.

  * Legacy helpers to implement all the legacy driver entry points
    (page_flip, set_config, plane vfuncs) on top of the new atomic driver
    interface. These are completely driver agnostic. The reason for having
    the legacy support as helpers is that drivers can switch step-by-step.
    And they could e.g. even keep the legacy page_flip code around for some
    old platforms where converting to full-blown atomic isn't worth it.

  * Atomic helpers which implement the various new ->atomic_* driver
    interfaces in terms of the revised crtc helper and new plane helper
    hooks.

- The revised crtc helper implemenation essentially implements all the
  lessons learned in the i915 modeset rework (when using the atomic helpers
  only):

  * Enable/disable sequence for a given config are always the same and
    callbacks are always called in the same order. This contrast starkly
    with the crtc helpers, where the sequence of operations is heavily
    dependent on the previous config.

    One corollary of this is that if the configuration of a crtc only
    partially changes (e.g. a connector moves in a cloned config) the
    helper code will still disable/enable the full display pipeline. This
    is the only way to ensure that the enable/disable sequence is always
    the same.

  * It won't call disable or enable hooks more than once any more because
    it lost track of state, thanks to the atomic state tracking. And if
    drivers implement the ->reset hook properly (by either resetting the hw
    or reading out the hw state into the atomic structures) this even
    extends to the hardware state. So no more disable-me-harder kind of
    nonsense.

  * The only thing missing is the hw state readout/cross-check support, but
    if drivers have hw state readout support in their ->reset handlers it's
    simple to extend that to cross-check the hw state.

  * The crtc->mode_set callback is gone and its replacement only sets crtc
    timings and no longer updates the primary plane state. This way we can
    finally implement primary planes properly.

- The new plane helpers should be suitable enough for pretty much
  everything, and a perfect fit for hardware with GO bits. Even if they
  don't fit the atomic helper library is rather flexible and exports all
  the functions for the individual steps to drivers. So drivers can pick
  what matches and implement their own magic for everything else.

- A big difference compared to all previous atomic series is that this one
  doesn't implement async commit in a generic way. Imo driver requirements
  for that are too diverse to create anything reasonable sane which would
  actually work on a reasonable amount of different drivers. Also, we've
  never had a helper library for page_flips even, so it's really hard to
  know what might work and what's stupid without a bit of experience in the form
  of a few driver implementations.

  I think with the current flexibility for drivers to pick individual
  stages and existing helpers like drm_flip_queue it's rather easy though
  to implement proper async commit.

- There's a few other differences of minor importance to earlier atomic
  series:

  * Common/generic properties are parsed in the callers/core and not in
    drivers, and passed to drivers by directly setting the right members in
    atomic state structures. That greatly simplifies all the transitional
    and legacy helpers an removes a lot of boilerplate code.

  * There's no crazy trylock mode used for the async commit since these
    helpers don't do async commit. A simple ordered flip queue of atomic
    state updates should be sufficient for preventing concurrent hw access
    anyway, as long as synchronous updates stall correctly with e.g.
    flush_work_queue or similar function. Abusing locks to enforce ordering
    isn't a good idea imo anyway.

  * These helpers reuse the existing ->mode_fixup hooks in the atomic_check
    callback. Which means that drivers need to adapat and move a lot less code
    into their atomic_check callbacks.

Now this isn't everything needed in the drm core and helpers for full
atomic support. But it's enough to start with converting drivers, and
except for actually testing multiplane and multicrtc updates also enough to
implement full atomic updates. Still missing are:

- Per-plane locking. Since these helpers here encapsulate the locking
  completely this should be fairly easy to implement.

- fbdev support for atomic_check/commit, so that multi-pipe finally works
  sanely in fbcon.

- Adding and decoding shared/core properties. That just needs to be rebased
  from Rob's latest patch series, with minor adjustments so that the
  decoding happens in the core instead of in drivers.

- Actually adding the atomic ioctl. Again just rebasing Rob's latest patch
  should be all that's needed.

- Resolving how to deal with DPMS in atomic. Atomic is a good excuse to fix up
  the crazy semantics dpms currently has. I'm floating an RFC about this topic
  already.

- Finally I couldn't test connector/encoder stealing properly since my test
  vehicle here doesn't allow a connector on different crtcs. So drivers
  which support this might see some surprises in that area. There is no semantic
  change though in how encoder stealing and assignment works (or at least no
  intended one), so I think the risk is minimal.

As just mentioned I've done a fake conversion of an existing driver using
crtc helpers to debug the helper code and validate the smooth transition
approach. And that smooth transition was the really big motivation for
this. It seems to actually work and consists of 3 phases:

Phase 1: Rework driver backend for crtc/plane helpers

The requirement here is that universal plane support is already implement. If
universal plane support isn't implement yet it might be better though to just do
it as part of this phase, directly using the new plane helpers. There are two
big things to do:

- Split up the existing ->update/disable_plane hooks into check/commit
  hooks and extract the crtc-wide prep/flush parts (like setting/clearing
  GO bits).

- The other big change is to split the crtc->mode_set hook into the plane
  update (done using the plane helpers) and the crtc setup in a new
  ->mode_set_nofb hook.

When phase 1 is complete the driver implements all the new callbacks which
push the software state into hardware, but still using all the legacy entry
points and crtc helpers. The transitional helpers serve as impendance
mismatch here.

Phase 2: Rework state handling

This consists of rolling out the state handling helpers for planes, crtcs
and connectors and reviewing all ->mode_fixup and similar hooks to make
sure they don't depend upon implicit global state which might change in the
atomic world. Any such code must be moved into ->atomic_check functions which
just rely on the free-standing atomic state update structures.

This phase also adds a few small pieces of fixup code to make sure the
atomic state doesn't get out of sync in the legacy driver callbacks.

Phase 3: Roll out atomic support

Now it's just about replacing vfuncs with the ones provided by the helper
and filling out the small missing pieces (like atomic_check logic or async
commit support needed for page_flips). Due to the prep work in phase 1 no
changes to the driver backend functions should be required, and because of
the prep work in phase 2 atomic implementations can be rolled out
step-by-step. So if async commit ins't implemented yet page_flip can be
implemented with the legacy functions without wreaking havoc in the other
operations.

* tag 'topic/atomic-helpers-2014-11-09' of git://anongit.freedesktop.org/drm-intel:
  drm/atomic: Refcounting for plane_state->fb
  drm: Docbook integration and over sections for all the new helpers
  drm/atomic-helpers: functions for state duplicate/destroy/reset
  drm/atomic-helper: implement ->page_flip
  drm/atomic-helpers: document how to implement async commit
  drm/atomic: Integrate fence support
  drm/atomic-helper: implementatations for legacy interfaces
  drm: Atomic crtc/connector updates using crtc/plane helper interfaces
  drm/crtc-helper: Transitional functions using atomic plane helpers
  drm/plane-helper: transitional atomic plane helpers
  drm: Add atomic/plane helpers
  drm: Global atomic state handling
  drm: Add atomic driver interface definitions for objects
  drm/modeset_lock: document trylock_only in kerneldoc
  drm: fixup kerneldoc in drm_crtc.h
  drm: Pull drm_crtc.h into the kerneldoc template
  drm: Move drm_crtc_init from drm_crtc.h to drm_plane_helper.h

Pull arm64 fixes from Catalin Marinas:
 - enable bpf syscall for compat
 - cpu_suspend fix when checking the idle state type
 - defconfig update

* tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux:
  arm64: defconfig: update defconfig for 3.18
  arm64: compat: Enable bpf syscall
  arm64: psci: fix cpu_suspend to check idle state type for index

Pull ARM SoC fixes from Olof Johansson:
 "Another quiet week:

   - a fix to silence edma probe error on non-supported platforms from
     Arnd
   - a fix to enable the PL clock for Parallella, to make mainline
     usable with the SDK.
   - a somewhat verbose fix for the PLL clock tree on VF610
   - enabling of SD/MMC on one of the VF610-based boards (for testing)
   - a fix for i.MX where CONFIG_SPI used to be implicitly enabled and
     now needs to be added to the defconfig instead
   - another maintainer added for bcm2835: Lee Jones"

* tag 'armsoc-for-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc:
  ARM: dts: zynq: Enable PL clocks for Parallella
  dma: edma: move device registration to platform code
  ARM: dts: vf610: add SD node to cosmic dts
  MAINTAINERS: update bcm2835 entry
  ARM: imx: Fix the removal of CONFIG_SPI option
  ARM: imx: clk-vf610: define PLL's clock tree

Pull devicetree bugfix from Grant Likely:
 "One buffer overflow bug that shouldn't be left around"

* 'devicetree/merge' of git://git.kernel.org/pub/scm/linux/kernel/git/glikely/linux:
  of: Fix overflow bug in string property parsing functions

Pull btrfs fix from Chris Mason:
 "It's a one liner for an error cleanup path that leads to crashes"

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs:
  Btrfs: fix kfree on list_head in btrfs_lookup_csums_range error cleanup

Merge tag 'driver-core-3.18-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core

Pull driver core fixes from Greg KH:
 "Here are 3 tiny fixes for 3.18-rc4.

  One fixes up a long-stading race condition in the driver core for
  removing directories in /sys/devices/virtual/ and the other 2 fix up
  the wording of a new Kconfig option that was added in 3.18-rc1"

* tag 'driver-core-3.18-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core:
  tiny: rename ENABLE_DEV_COREDUMP to ALLOW_DEV_COREDUMP
  tiny: reverse logic for DISABLE_DEV_COREDUMP
  sysfs: driver core: Fix glue dir race condition by gdp_mutex