Not all registers need forcewake even if they're not shadowed.
Add the missing check to gen8_writeX() to avoid needless forcewake
usage when writing eg. display registers.

v2: Use straight up <0x40000 check instead of NEEDS_FORCE_WAKE()
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The BIOS or someone else might have done something bad and there
might be old GT FIFO erros reported in GTFIFODBG. Clear those out
in intel_uncore_early_sanitize() to make sure we don't mistake them
for our problems.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

These are needed when we cat the debugfs and sysfs files.

V2: - Rebase
V3: - Rebase
V4: - Rebase
Signed-off-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NRodrigo Vivi <rodrigo.vivi@gmail.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This patch adds the initial infrastructure to allow a Runtime PM
implementation that sets the device to its D3 state. The patch just
adds the necessary callbacks and the initial infrastructure.

We still don't have any platform that actually uses this
infrastructure, we still don't call get/put in all the places we need
to, and we don't have any function to save/restore the state of the
registers. This is not a problem since no platform uses the code added
by this patch. We have a few people simultaneously working on runtime
PM, so this initial code could help everybody make their plans.

V2: - Move some functions to intel_pm.c
    - Remove useless pm_runtime_allow() call at init
    - Remove useless pm_runtime_mark_last_busy() call at get
    - Use pm_runtime_get_sync() instead of 2 calls
    - Add a WARN to check if we're really awake

V3: - Rebase.

V4: - Don't need to call pci_{save,restore}_state and
      pci_set_power_sate, since they're already called by the PCI
      layer
    - Remove wrong pm_runtime_enable() call at init_runtime_pm
Signed-off-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NRodrigo Vivi <rodrigo.vivi@gmail.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Since early sanitize and uncore sanitize are called one after the other,
I think, we can remove second forcewake reset which was are calling
twice in both the functions.

Note that this is merge fallout between

commit ef46e0d2
Author: Daniel Vetter <daniel.vetter@ffwll.ch>
Date:   Sat Nov 16 16:00:09 2013 +0100

    drm/i915: restore the early forcewake cleanup

and

commit 521198a2
Author: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date:   Fri Aug 23 16:52:30 2013 +0300

    drm/i915: sanitize forcewake registers on reset
Signed-off-by: NDeepak S <deepak.s@intel.com>
[danvet: Explain how this came to be.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

On VLV, FIFO will be shared by both SW and HW. So, we read the
free entries through register and update dev_priv variable
and wait for only 20 entries to be free

From Deepak's follow-up mail explaining why vlv is special:

"On SB, Out of 64 FIFO Entries, 20 Entries will be used by HW and
remaining 44 will be used by the SW,. I think due to this reason, we
have a threshold of 20 Entries."

"On VLV, HW and SW can access all 64 fifo entries, I don't think
having a threshold of 20 Entries is mandatory on VLV. Also, since both
SW and HW can access all 64 Entries. I think on VLV, we need to update
the fifo_count before waiting for the FIFO."

v2: Apply mask when we read the number of free FIFO entries (Ville).

v3: Mask applied after reading the register (Deepak).
Signed-off-by: NDeepak S <deepak.s@intel.com>
[danvet: Add further explanation from Deepak to commit message.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Add a REG_WRITE_FOOTER macro as a counterpart to the REG_WRITE_HEADER.
The current code has the spin_lock() in the HEADER, but the
spin_unlock() is open coded, which looks rather confusing on the first
glance. A bit of additional symmetry might help.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

It doesn't like that we assign 0 to a pointer, it wants the real NULL.

On closer look that initialization is actually bogus, and the compiler
can easily see that we never use it unitialized. So let's just drop
this.

Cc: Deepak S <deepak.s@intel.com>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Reported-by: Nkbuild test robot <fengguang.wu@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This was fumbled in the conversion to per-engine forcewake.

Cc: Deepak S <deepak.s@intel.com>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

On VLV the GTFIFOCTL register has other bits besides the number of free
entries in the GT wake FIFO. Apply a mask when we read th register to
make sure we don't misinterpret the number of free FIFO entries.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
[danvet: There's some unclarity about hsw, but brushed off as todays'
Bspec just acting up a bit.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

On VLV GTFIFODBG has more bits. Just report them all.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Split vlv force wake routines to help individually control Media/Render
well based on the register access.

We've seen power savings in the lower sub-1W range on workloads that
only need on of the power wells, e.g. glbenchmark, media playback

Note: The same split isn't there for the forcewake queue, only the
forcwake domains are split.
Signed-off-by: NDeepak S <deepak.s@intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
[danvet: Rebase on top of the removed forcewake hack in the ring irq
get/put code and add a note to add Deepak's answer to Chris question.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Added power well arguments to all the force wake routines
to help us individually control power well based on the
scenario.
Signed-off-by: NDeepak S <deepak.s@intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
[danvet: Resolve conflict with the removed forcewake hack and drop one
spurious hunk Jesse noticed.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This was forgotten in

commit 9d1cb914
Author: Paulo Zanoni <paulo.r.zanoni@intel.com>
Date:   Fri Nov 1 13:32:08 2013 -0200

    drm/i915: avoid unclaimed registers when capturing the error state

Cc: Paulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Some BIOS just leak the forcewak bits, which we clean up.
Unfortunately this has been broken in

commit 521198a2
Author: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date:   Fri Aug 23 16:52:30 2013 +0300

    drm/i915: sanitize forcewake registers on reset

To make this work both for resets and for BIOS takeover just add the
forcewake clearing call back to intel_uncore_early_sanitize.

We need to clear the forcewake in early sanitize so that the forcewak
dance in intel_uncore_init (to figure out whether we have mt or legacy
forcewake on ivb) works. That cleanup fits in nicely with the general
topic of early_sanitize to prepare for the very first mmio ops.

Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Reported-by: NJörg Otte <jrg.otte@gmail.com>
Cc: Jörg Otte <jrg.otte@gmail.com>
References: https://lkml.org/lkml/2013/11/16/40
Cc: stable@vger.kernel.org (for 3.12 only)
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This patch existed before, but was lost over time.

Note that reset is still somewhat problematic in my limited testing (ie.
module_reload will not pass) but it can be disabled with a module
parameter, and support should be considered preliminary anyway.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Insist that flags and pad fields are zero, so that
we can safely extend the interface in future.

Testcase: igt/gem_reset_stats/params
Suggested-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NMika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This ioctl returns reset stats for specified context.

The struct returned contains context loss counters.

reset_count:    all resets across all contexts
batch_active:   active batches lost on resets
batch_pending:  pending batches lost on resets

v2: get rid of state tracking completely and deliver only counts. Idea
    from Chris Wilson.

v3: fix commit message

v4: default context handled inside i915_gem_context_get_hang_stats

v5: reset_count only for priviledged process

v6: ctx=0 needs CAP_SYS_ADMIN for batch_* counters (Chris Wilson)

v7: context hang stats never returns NULL

v8: rebased on top of reworked context hang stats
    DRM_RENDER_ALLOW for ioctl

v9: use DEFAULT_CONTEXT_ID. Improve comments for ioctl struct members
Signed-off-by: NMika Kuoppala <mika.kuoppala@intel.com>
Cc: Ian Romanick <idr@freedesktop.org>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Reviewed-by: NIan Romanick <ian.d.romanick@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

It's no longer a required workaround on BDW.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
[danvet: Move compile fix from a later patch to this one.]
Reviewed-by: NJani Nikula <jani.nikula@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Somehow this got missed or dropped during development. The simulator
does not use forcewake, so it's entirely possible it never worked
correctly. After the mmio rework, this will end up in an OOPs, and the
system will not boot.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
[danvet: Use IS_GEN8 instead of IS_BROADWELL.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

GEN8 removes the GT FIFO which we've all come to know and love. Instead
it offers a wider range of optimized registers which always keep a
shadowed copy, and are fed to the GPU when it wakes.

How this is implemented in hardware is still somewhat of a mystery. As
far as I can tell, the basic design is as follows:

If the register is not optimized, you must use the old forcewake
mechanism to bring the GT out of sleep. [1]

If register is in the optimized list the write will signal that the
GT should begin to come out of whatever sleep state it is in.

While the GT is coming out of sleep, the requested write will be stored
in an intermediate shadow register.

Do to the fact that the implementation details are not clear, I see
several risks:
1. Order is not preserved as it is with GT FIFO. If we issue multiple
writes to optimized registers, where order matters, we may need to
serialize it with forcewake.
2. The optimized registers have only 1 shadowed slot, meaning if we
issue multiple writes to the same register, and those values need to
reach the GPU in order, forcewake will be required.

[1] We could implement a SW queue the way the GT FIFO used to work if
desired.

NOTE: Compile tested only until we get real silicon.

v2:
- Use a default case to make future platforms also work.
- Get rid of IS_BROADWELL since that's not yet defined, but we want to
  MMIO as soon as possible.

v3: Apply suggestions from Mika's review:
- s/optimized/shadowed/
- invert the logic of the helper so that it does what it says (the
  code itself was correct, just confusing to read).

v4:
- Squash in lost break.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Now that MMIO has been split up into gen specific functions it is
obvious when HAS_FPGA_DBG_UNCLAIMED, HAS_FORCE_WAKE are needed. As such,
we can remove this extraneous condition.

As a result of this, as well as previously existing function pointers
for forcewake, we no longer need the has_force_wake member in the device
specific data structure.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Similar to the previous patch which implemented GEN specific reads; this
patch does the same for writes. Writes have a bit of adding complexity
due to the FPGA_DBG feature of HSW plus:

gen[2-4]: nothing special
gen5: ILK dummy write
gen[6-7]: forcewake shenanigans
gen[HSW}: forcewake shenanigans + FPGA_DBG

I was a bit torn about whether or not to combine 6-HSW as one function,
since the FPGA_DBG is cleanly separated, and it wouldn't make the 6-7
MMIO too messy. In the end, I chose the clearest possible solution which
splits out HSW.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Extracting the MMIO read functionality makes per gen handling a bit
simpler, and the overall function a lot easier to read. The increasing
complexity of reads doesn't get too bad as the generation number
increases:

gen[2-4]: Nothing special
gen5: ILK dummy write workaround
gen6+: forcewake shenanigans
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Just to make the churn and code duplication in upcoming patches a bit
less, turn code which is common to all GEN MMIO functions into a macro.

v2: Fix typo in subject
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

In preparation for having per GEN MMIO functions, create, and start
using MMIO functions in our uncore data structure. This simply makes the
transition easier by allowing us to just plug in the per GEN stuff
later.

For simplicity, I moved the intel_uncore_init() function down since
those rely on static functions defined lower in the file. This is most
of the churn in this patch.

I made one unrelated change here by using off_t datatype for the offset
of the register to write. I like the clarity that this brings to the
code. If I did it as a separate patch, I am pretty certain it would get
bikeshedded to oblivion.
Requested-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

In order to be able to have virtual functions for the MMIO, we need to
use the raw access function. To keep things simple, just move this to
our early_sanitize code in uncore.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

At least on my i830M here it reliably results in hard system hangs
nowadays. This is much worse than falling back to software rendering,
so I think we should simply rip this out.

After all we don't have any gpu reset for gen3 either, and there are a
lot more of those still around.

Cc: Chris Wilson <chris@chris-wilson.co.uk>
Acked-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

During system boot up, by default, the power gate for render, media and
display well still power gated.  Normally, BIOS will turn off the power
gate.  In the BIOS-less system, the driver need to turn off the power
gate very early during driver load.

v2: Move this to intel_uncore_sanitize to allow it to get call during
resume path. (Daniel)
v3: Remove redundant write 0 to DPIO_CTL, and use DPIO_RESET instead of
just 0x1 (Ville)
    Add turn of power gate for display 2d/render well/media well.
v4: Remove toggle cmnreset in intel_uncore_sanitize.  Cmnreset should
toggle after CRI clock source has been selected.  Jesse DPIO reset patch
which toggle the cmnreset in intel_modeset_init_hw() should handle it.
(Ville)
Signed-off-by: NChon Ming Lee <chon.ming.lee@intel.com>
Reviewed-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Obtaining the forcwake requires expensive and time consuming
serialisation. And we often try to obtain the forcewake multiple times
in very quick succession. We can reduce the overhead of these sequences
by delaying the forcewake release, and so not hammer the hw quite so
hard.

I was hoping this would help with the spurious
[drm:__gen6_gt_force_wake_mt_get] *ERROR* Timed out waiting for forcewake old ack to clear.
found on Haswell. Alas not.

v2: Fix teardown ordering - unmap the regs after turning off forcewake,
and make sure we do turn off forcewake - both found by Ville.

v3: As we introduce intel_uncore_fini(), use it to make sure everything
is disabled before we hand back to the BIOS.

Note: I have no claims for improved performance, stablity or power
comsumption for this patch. We should not be hitting the registers often
enough for this to improve benchmarks, but given the nature of our hw it
is likely to improve long term stability.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

In reset we try to restore the forcewake state to
pre reset state, using forcewake_count. The reset
doesn't seem to clear the forcewake bits so we
get warn on forcewake ack register not clearing.

Use same mechanism as intel_uncore_sanitize() does
when loading driver to reset the forcewake bits, right
after the chip has been reset.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NMika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The TRACE_EVENT_CONDITION is supposed to generate more efficient code
than if (cond) trace(), which is what we are currently using inside the
register access functions.

v2: Rebase onto uncore
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The INTEL_INFO() macro extracts the dev_private pointer from the device,
so passing in the dev_private->dev is a long winded circumlocution.

v2: rebase onto uncore
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Detangle the confusion that NOTRACE variants of the register read/write
routines were directly using the raw register access. We need for those
routines to reuse the common code for serializing register access and
ensuring the correct register power states. This is only possible now
that the only routines that required raw access use their own API.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The GT functions for enabling register access also need to occasionally
write to and read from registers. To avoid the potential recursion as we
modify the public interface to be stricter, introduce a private register
access API for the GT functions.

v2: Rebase
v3: Rebase onto uncore
v4: Use raw interfaces consistently so that we only use the low-level
    readN functions from a single location.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Currently, the register access code is split between i915_drv.c and
intel_pm.c. It only bares a superficial resemblance to the reset of the
powermanagement code, so move it all into its own file. This is to ease
further patches to enforce serialised register access.

v2: Scan for random abuse of I915_WRITE_NOTRACE
v3: Take the opportunity to rename the GT functions as uncore. Uncore is
the term used by the hardware design (and bspec) for all functions
outside of the GPU (and CPU) cores in what is also known as the System
Agent.
v4: Rebase onto SNB rc6 fixes
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
[danvet: Wrestle patch into applying and inline
intel_uncore_early_sanitize (plus move the old comment to the new
function). Also keep the _santize postfix for intel_uncore_sanitize.]
[danvet: Squash in fixup spotted by Chris on irc: We need to call
intel_pm_init before intel_uncore_sanitize since the later will call
cancel_work on the delayed rps setup work the former initializes.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>