This is a regression introduced in

commit 0294ae7b
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Thu Mar 13 12:00:29 2014 +0000

    drm/i915: Consolidate forcewake resetting to a single function

The reordered setup sequence ended up calling del_timer_sync before
the timer was set up correctly, resulting in endless hilarity when
loading the driver.

Compared to Ben's patch (which moved around the setup_timer call to
sanitize_early) this moves the sanitize_early call around in the
driver load call. This way we avoid calling setup_timer again in the
resume code (where we also call sanitize_early).

Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Tested-by: NRodrigo Vivi <rodrigo.vivi@gmail.com>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=76242Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We have two paths that try to reset the forcewake registers back to
known good values, with slightly different semantics and levels of
paranoia. Combine the two by passing a parameter to either restore the
forcewake status or to clear our bookkeeping, and raise the paranoia
level to max.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Now that we regularly defer the forcewake dance to a timer func, it is
likely to fire after we disable the device during suspend. This
generates an oops as we detect inconsistency in the hardware state. So
before suspend, we want to complete the outstanding dance and generally
sanitize the registers before handing back to the BIOS.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

In the upcoming patches we'll need to access the rest of the fields in
the punit power gating register, so prepare for that.

v2:
- add doc reference for the power well subsystem IDs (Jesse)
- remove IDs for non-existant DPIO_RX[23] subsystems (Jesse)
Signed-off-by: NImre Deak <imre.deak@intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

As we now have intel_uncore_forcewake_reset() no need
to do explicit put after reset.

v2: rebase
Signed-off-by: NMika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

as they don't exists.

v2: rename gen6_*_mt_* to gen7_*_mt_* as they never get called
    with gen6 (Chris)
Signed-off-by: NMika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

When we get control from BIOS there might be mt forcewake
bits already set. This causes us to do double mt get
without proper clear/ack sequence.

Fix this by clearing mt forcewake register on init,
like we do with older gens.
Signed-off-by: NMika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

It occured to me that when we're trying to wake up both render
and media wells on VLV, we might end up calling the low level
force_wake_get/put two times even though one call would be
enough. Make that happen by figuring out which wells really
need to be woken up based on the forcewake counts.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NDeepak S <deepak.s@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

VLV is the only platform where we increment/decrement the forcewake
count around register access. Drop the inc/dec on VLV to make the
forcewake code a bit more unified.

The inc/dec are not necessary since we hold the uncore lock around
the whole operation.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NDeepak S <deepak.s@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Use the render/media specific forcewake counts to properly restore the
forcewake status after a GPU reset on VLV.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NDeepak S <deepak.s@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

I could swear this was already happening in the current code...

Also, put the reads and writes in a generic place, so we don't forget
it again when we add runtime PM support to new platforms.
Signed-off-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NImre Deak <imre.deak@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Just to be sure...
Signed-off-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NImre Deak <imre.deak@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Because we shouldn't be runtime suspended when forcewake is supposed
to be enabled.
Signed-off-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NImre Deak <imre.deak@intel.com>
[danvet: Update commit message - no WARN expected since the bugfix for
issues hit with this assert is already in. And resolve conflicts with
the change from worker to timer for the delayed fw release.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

When we call gen6_gt_force_wake_put we don't actually put force_wake,
we just schedule gen6_force_wake_work through mod_delayed_work, and
that will eventually release force_wake.

The problem is that we call intel_runtime_pm_put directly at
gen6_gt_force_wake_put, so most of the times we put our runtime PM
reference before the delayed work happens, so we may runtime suspend
while force_wake is still supposed to be enabled if the graphics
autosuspend_delay_ms is too small.

Now the nice thing about the current code is that after it triggers
the delayed work function it gets a refcount, and it only triggers the
delayed work function if refcount is zero. This guarantees that when
we schedule the funciton, it will run before we try to schedule it
again, which simplifies the problem and allows for the current
solution to work properly (hopefully!).

v2: - Keep the VLV refcounts balanced (Jesse)
Signed-off-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We don't want to suffer scheduling delay when turning off the GPU after
waking it up to touch registers. Ideally, we only want to keep the GPU
awake for the register access sequence, with a single forcewake dance on
the first access and release immediately after the last. We set a timer
on the first access so that we only dance once and on the next scheduler
tick, we drop the forcewake again.

This moves the cleanup routine from the common i915 workqueue to a timer
func so that we don't anger powertop, and drop the forcewake again
quicker.

v2: Enable the deferred force_wake_put for regular register reads as
    well.
v3: Beautification and make sure we disable forcewake when shutting
    down.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Ben Widawsky <ben@bwidawsk.net>
Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Sometimes generic driver code gets forcewake explicitly by
gen6_gt_force_wake_get(), which check forcewake_count before accessing
hardware. However the register access with gen8_write function access
low level hw accessors directly, ignoring the forcewake_count. This
leads to nested forcewake get from hardware, in ring init and possibly
elsewhere, causing forcewake ack clear errors and/or hangs.

Fix this by checking the forcewake count also in gen8_write

v2: Read side doesn't care about shadowed registers,
    Remove __needs_put funkiness from gen8_write. (Ville)
    Improved commit message.

References: https://bugs.freedesktop.org/show_bug.cgi?id=74007Signed-off-by: NMika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: NMika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Nothing's changed here; we just need to bump the generation check.
Signed-off-by: NKenneth Graunke <kenneth@whitecape.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We need to have the address space when reserving space for the objects.
Since the address space and context are tied together, and reserve
occurs before context switch (for good reason), we must lookup our
context earlier in the process.

This leaves some room for optimizations where we no longer need to use
ctx_id in certain places. This will be addressed in a subsequent patch.

Important tricky bit:
Because slow relocations during execbuffer drop struct_mutex

Perhaps it would be best to acquire the reference when we get the
context, but I'll save that for another day (note I have written the
patch before, and I found the changes required to be uglier than this).

Note that since we currently access everything via context id, and not
the data structure this is fine, though not desirable. The next change
attempts to get the context only once via the context ID idr lookup, and
as such, the following can happen:

CTX-A is created, refcount = 1
CTX-A execbuf, mutex dropped
close IOCTL called on CTX-A, refcount = 0
CTX-A resumes in execbuf.

v2: Rebased on top of
commit b6359918
Author: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date:   Wed Oct 30 15:44:16 2013 +0200

    drm/i915: add i915_get_reset_stats_ioctl

v3: Rebased on top of
commit 25b3dfc8
Author: Mika Westerberg <mika.westerberg@linux.intel.com>
Date:   Tue Nov 12 11:57:30 2013 +0200

Author: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date:   Tue Nov 26 16:14:33 2013 +0200

    drm/i915: check context reset stats before relocations
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Not all registers need forcewake even if they're not shadowed.
Add the missing check to gen8_writeX() to avoid needless forcewake
usage when writing eg. display registers.

v2: Use straight up <0x40000 check instead of NEEDS_FORCE_WAKE()
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The BIOS or someone else might have done something bad and there
might be old GT FIFO erros reported in GTFIFODBG. Clear those out
in intel_uncore_early_sanitize() to make sure we don't mistake them
for our problems.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

These are needed when we cat the debugfs and sysfs files.

V2: - Rebase
V3: - Rebase
V4: - Rebase
Signed-off-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NRodrigo Vivi <rodrigo.vivi@gmail.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This patch adds the initial infrastructure to allow a Runtime PM
implementation that sets the device to its D3 state. The patch just
adds the necessary callbacks and the initial infrastructure.

We still don't have any platform that actually uses this
infrastructure, we still don't call get/put in all the places we need
to, and we don't have any function to save/restore the state of the
registers. This is not a problem since no platform uses the code added
by this patch. We have a few people simultaneously working on runtime
PM, so this initial code could help everybody make their plans.

V2: - Move some functions to intel_pm.c
    - Remove useless pm_runtime_allow() call at init
    - Remove useless pm_runtime_mark_last_busy() call at get
    - Use pm_runtime_get_sync() instead of 2 calls
    - Add a WARN to check if we're really awake

V3: - Rebase.

V4: - Don't need to call pci_{save,restore}_state and
      pci_set_power_sate, since they're already called by the PCI
      layer
    - Remove wrong pm_runtime_enable() call at init_runtime_pm
Signed-off-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NRodrigo Vivi <rodrigo.vivi@gmail.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Since early sanitize and uncore sanitize are called one after the other,
I think, we can remove second forcewake reset which was are calling
twice in both the functions.

Note that this is merge fallout between

commit ef46e0d2
Author: Daniel Vetter <daniel.vetter@ffwll.ch>
Date:   Sat Nov 16 16:00:09 2013 +0100

    drm/i915: restore the early forcewake cleanup

and

commit 521198a2
Author: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date:   Fri Aug 23 16:52:30 2013 +0300

    drm/i915: sanitize forcewake registers on reset
Signed-off-by: NDeepak S <deepak.s@intel.com>
[danvet: Explain how this came to be.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

On VLV, FIFO will be shared by both SW and HW. So, we read the
free entries through register and update dev_priv variable
and wait for only 20 entries to be free

From Deepak's follow-up mail explaining why vlv is special:

"On SB, Out of 64 FIFO Entries, 20 Entries will be used by HW and
remaining 44 will be used by the SW,. I think due to this reason, we
have a threshold of 20 Entries."

"On VLV, HW and SW can access all 64 fifo entries, I don't think
having a threshold of 20 Entries is mandatory on VLV. Also, since both
SW and HW can access all 64 Entries. I think on VLV, we need to update
the fifo_count before waiting for the FIFO."

v2: Apply mask when we read the number of free FIFO entries (Ville).

v3: Mask applied after reading the register (Deepak).
Signed-off-by: NDeepak S <deepak.s@intel.com>
[danvet: Add further explanation from Deepak to commit message.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Add a REG_WRITE_FOOTER macro as a counterpart to the REG_WRITE_HEADER.
The current code has the spin_lock() in the HEADER, but the
spin_unlock() is open coded, which looks rather confusing on the first
glance. A bit of additional symmetry might help.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

It doesn't like that we assign 0 to a pointer, it wants the real NULL.

On closer look that initialization is actually bogus, and the compiler
can easily see that we never use it unitialized. So let's just drop
this.

Cc: Deepak S <deepak.s@intel.com>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Reported-by: Nkbuild test robot <fengguang.wu@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This was fumbled in the conversion to per-engine forcewake.

Cc: Deepak S <deepak.s@intel.com>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

On VLV the GTFIFOCTL register has other bits besides the number of free
entries in the GT wake FIFO. Apply a mask when we read th register to
make sure we don't misinterpret the number of free FIFO entries.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
[danvet: There's some unclarity about hsw, but brushed off as todays'
Bspec just acting up a bit.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

On VLV GTFIFODBG has more bits. Just report them all.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Split vlv force wake routines to help individually control Media/Render
well based on the register access.

We've seen power savings in the lower sub-1W range on workloads that
only need on of the power wells, e.g. glbenchmark, media playback

Note: The same split isn't there for the forcewake queue, only the
forcwake domains are split.
Signed-off-by: NDeepak S <deepak.s@intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
[danvet: Rebase on top of the removed forcewake hack in the ring irq
get/put code and add a note to add Deepak's answer to Chris question.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Added power well arguments to all the force wake routines
to help us individually control power well based on the
scenario.
Signed-off-by: NDeepak S <deepak.s@intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
[danvet: Resolve conflict with the removed forcewake hack and drop one
spurious hunk Jesse noticed.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This was forgotten in

commit 9d1cb914
Author: Paulo Zanoni <paulo.r.zanoni@intel.com>
Date:   Fri Nov 1 13:32:08 2013 -0200

    drm/i915: avoid unclaimed registers when capturing the error state

Cc: Paulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Some BIOS just leak the forcewak bits, which we clean up.
Unfortunately this has been broken in

commit 521198a2
Author: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date:   Fri Aug 23 16:52:30 2013 +0300

    drm/i915: sanitize forcewake registers on reset

To make this work both for resets and for BIOS takeover just add the
forcewake clearing call back to intel_uncore_early_sanitize.

We need to clear the forcewake in early sanitize so that the forcewak
dance in intel_uncore_init (to figure out whether we have mt or legacy
forcewake on ivb) works. That cleanup fits in nicely with the general
topic of early_sanitize to prepare for the very first mmio ops.

Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Reported-by: NJörg Otte <jrg.otte@gmail.com>
Cc: Jörg Otte <jrg.otte@gmail.com>
References: https://lkml.org/lkml/2013/11/16/40
Cc: stable@vger.kernel.org (for 3.12 only)
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This patch existed before, but was lost over time.

Note that reset is still somewhat problematic in my limited testing (ie.
module_reload will not pass) but it can be disabled with a module
parameter, and support should be considered preliminary anyway.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Insist that flags and pad fields are zero, so that
we can safely extend the interface in future.

Testcase: igt/gem_reset_stats/params
Suggested-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NMika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This ioctl returns reset stats for specified context.

The struct returned contains context loss counters.

reset_count:    all resets across all contexts
batch_active:   active batches lost on resets
batch_pending:  pending batches lost on resets

v2: get rid of state tracking completely and deliver only counts. Idea
    from Chris Wilson.

v3: fix commit message

v4: default context handled inside i915_gem_context_get_hang_stats

v5: reset_count only for priviledged process

v6: ctx=0 needs CAP_SYS_ADMIN for batch_* counters (Chris Wilson)

v7: context hang stats never returns NULL

v8: rebased on top of reworked context hang stats
    DRM_RENDER_ALLOW for ioctl

v9: use DEFAULT_CONTEXT_ID. Improve comments for ioctl struct members
Signed-off-by: NMika Kuoppala <mika.kuoppala@intel.com>
Cc: Ian Romanick <idr@freedesktop.org>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Reviewed-by: NIan Romanick <ian.d.romanick@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

It's no longer a required workaround on BDW.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
[danvet: Move compile fix from a later patch to this one.]
Reviewed-by: NJani Nikula <jani.nikula@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Somehow this got missed or dropped during development. The simulator
does not use forcewake, so it's entirely possible it never worked
correctly. After the mmio rework, this will end up in an OOPs, and the
system will not boot.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
[danvet: Use IS_GEN8 instead of IS_BROADWELL.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

GEN8 removes the GT FIFO which we've all come to know and love. Instead
it offers a wider range of optimized registers which always keep a
shadowed copy, and are fed to the GPU when it wakes.

How this is implemented in hardware is still somewhat of a mystery. As
far as I can tell, the basic design is as follows:

If the register is not optimized, you must use the old forcewake
mechanism to bring the GT out of sleep. [1]

If register is in the optimized list the write will signal that the
GT should begin to come out of whatever sleep state it is in.

While the GT is coming out of sleep, the requested write will be stored
in an intermediate shadow register.

Do to the fact that the implementation details are not clear, I see
several risks:
1. Order is not preserved as it is with GT FIFO. If we issue multiple
writes to optimized registers, where order matters, we may need to
serialize it with forcewake.
2. The optimized registers have only 1 shadowed slot, meaning if we
issue multiple writes to the same register, and those values need to
reach the GPU in order, forcewake will be required.

[1] We could implement a SW queue the way the GT FIFO used to work if
desired.

NOTE: Compile tested only until we get real silicon.

v2:
- Use a default case to make future platforms also work.
- Get rid of IS_BROADWELL since that's not yet defined, but we want to
  MMIO as soon as possible.

v3: Apply suggestions from Mika's review:
- s/optimized/shadowed/
- invert the logic of the helper so that it does what it says (the
  code itself was correct, just confusing to read).

v4:
- Squash in lost break.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Now that MMIO has been split up into gen specific functions it is
obvious when HAS_FPGA_DBG_UNCLAIMED, HAS_FORCE_WAKE are needed. As such,
we can remove this extraneous condition.

As a result of this, as well as previously existing function pointers
for forcewake, we no longer need the has_force_wake member in the device
specific data structure.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>