When invalidating the TLBs it is documentated as requiring a post-sync
write. Failure to do so seems to result in a GPU hang.

Exposure to this hang on IVB seems to be a result of removing the extra
stalls required for SNB pipecontrol workarounds:

commit 6c6cf5aa
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Fri Jul 20 18:02:28 2012 +0100

    drm/i915: Only apply the SNB pipe control w/a to gen6

Note: Manually switch the pipe_control cmd to 4 dwords to avoid a
(silent) functional conflict with -next. This way will get a loud (but
conflict with next (since the scratch_addr has been deleted there).

Reported-and-tested-by: yex.tian@intel.com
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=53322Acked-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
[danvet: added note about merge conflict with -next.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We may only start to set up the new register values after having
confirmed that the ring is truely off. Otherwise the hw might lose the
newly written register values. This is caught later on in the init
sequence, when we check whether the register writes have stuck.

Cc: stable@vger.kernel.org
Reviewed-by: NJani Nikula <jani.nikula@intel.com>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=50522Tested-by: NYang Guang <guang.a.yang@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Having had to dive into the bspec to understand what each stage of the
workaround meant, and how that the ring broadcasting IDLE corresponded
with the GT powering down the ring (i.e. rc6) add comments to aide
the next reader.

And since the register "is used to control all aspects of PSMI and power
saving functions" that makes it quite interesting to inspect with
regards to RC6 hangs, so add it to the error-state.

v2: Rediscover the piece of magic, set the RNCID to 0 before waiting for
the ring to wake up.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The issue with this check is that it results in userspace receiving an
-EIO while the gpu reset hasn't completed, resulting in fallback to sw
rendering or worse.

Now there's also a stern comment in intel_ring_wait_seqno saying that
intel_ring_begin should not return -EAGAIN, ever, because some callers
can't handle that. But after an audit of the callsites I don't see any
issues. I guess the last problematic spot disappeared with the removal
of the pipelined fencing code.

So do the right thing and call check_wedge, which should properly
decide whether an -EAGAIN or -EIO is appropriate if wedged is set.

Note that the early check for a wedged gpu before touching the ring is
rather important (and it took me quite some time of acting like the
densest doofus to figure that out): If we don't do that and the gpu
died for good, not having been resurrect by the reset code, userspace
can merrily fill up the entire ring until it notices that something is
amiss.

Allowing userspace to emit more render, despite that we know that it
will fail can't lead to anything good (and by experience can lead to
all sorts of havoc, including angering the OOM gods and hard-hanging
the hw for good).

v2: Fix EAGAIN mispell, noticed by Chris Wilson.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Tested-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

So don't return -EAGAIN, even in the case of a gpu hang. Remap it to
-EIO instead. Note that this isn't really an issue with
interruptability, but more that we have quite a few codepaths (mostly
around kms stuff) that simply can't handle any errors and hence not
even -EAGAIN. Instead of adding proper failure paths so that we could
restart these ioctls we've opted for the cheap way out of sleeping
non-interruptibly.  Which works everywhere but when the gpu dies,
which this patch fixes.

So essentially interruptible == false means 'wait for the gpu or die
trying'.'

This patch is a bit ugly because intel_ring_begin is all non-interruptible
and hence only returns -EIO. But as the comment in there says,
auditing all the callsites would be a pain.

To avoid duplicating code, reuse i915_gem_check_wedge in __wait_seqno
and intel_wait_ring_buffer. Also use the opportunity to clarify the
different cases in i915_gem_check_wedge a bit with comments.

v2: Don't access dev_priv->mm.interruptible from check_wedge - we
might not hold dev->struct_mutex, making this racy. Instead pass
interruptible in as a parameter. I've noticed this because I've hit a
BUG_ON(!mutex_is_locked) at the top of check_wedge. This has been
added in

commit b4aca010
Author: Ben Widawsky <ben@bwidawsk.net>
Date:   Wed Apr 25 20:50:12 2012 -0700

    drm/i915: extract some common olr+wedge code

although that commit is missing any justification for this. I guess
it's just copy&paste, because the same commit add the same BUG_ON
check to check_olr, where it indeed makes sense.

But in check_wedge everything we access is protected by other means,
so this is superflous. And because it now gets in the way (we add a
new caller in __wait_seqno, which can be called without
dev->struct_mutext) let's just remove it.

v3: Group all the i915_gem_check_wedge refactoring into this patch, so
that this patch here is all about not returning -EAGAIN to callsites
that can't handle syscall restarting.

v4: Add clarification what interuptible == fales means in our code,
requested by Ben Widawsky.

v5: Fix EAGAIN mispell noticed by Chris Wilson.
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Tested-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The prep to remove the flushing list in

commit cc889e0f
Author: Daniel Vetter <daniel.vetter@ffwll.ch>
Date:   Wed Jun 13 20:45:19 2012 +0200

    drm/i915: disable flushing_list/gpu_write_list

causes quite some decent regressions. We can fix this by setting the
CS_STALL bit to ensure that the following seqno write happens only
after the cache flush has completed. But only do that when the caller
actually wants the flush (and not also when we invalidate caches
before starting the next batch).

I've looked through all our ancient scrolls about gen6+ pipe control
workarounds, and this seems to be indeed a legal combination: We're
allowed to set the CS_STALL bit when we flush the render cache (which
we do).

While yelling at this code, also pass back the return value from
intel_emit_post_sync_nonzero_flush properly.

v2: Instead of emitting more pipe controls, set the CS_STALL bit on
the write flush as suggested by Chris Wilson. It seems to work, too.

Cc: Eric Anholt <eric@anholt.net>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=51436
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=51429Tested-by: NLu Hua <huax.lu@intel.com>
Tested-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

From http://intellinuxgraphics.org/documentation/SNB/IHD_OS_Vol1_Part3.pdf

[DevSNB] If Flush TLB invalidation Mode is enabled it's the driver's
responsibility to invalidate the TLBs at least once after the previous
context switch after any GTT mappings changed (including new GTT
entries).  This can be done by a pipelined PIPE_CONTROL with TLB inv bit
set immediately before MI_SET_CONTEXT.

On GEN7 the invalidation mode is explicitly set, but this appears to be
lacking for GEN6. Since I don't know the history on this, I've decided
to dynamically read the value at ring init time, and use that value
throughout.

v2: better comment (daniel)
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>

This has showed up in several other patches. It's required for the next
context workaround.

I tested this one on its own and saw no differences in basic tests
(performance or otherwise). This patch is relatively likely to cause
regressions, hence why it's split out.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>

For that to work we need to export the base address of the gtt
mmio window from intel-gtt. Also replace all other uses of
dev->agp by values we already have at hand.
Reviewed-by: NJani Nikula <jani.nikula@linux.intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Empirical evidence suggests that we need to: On at least one ivb
machine when running the hangman i-g-t test, the rings don't properly
initialize properly - the RING_START registers seems to be stuck at
all zeros.

Holding forcewake around this register init sequences makes chip reset
reliable again. Note that this is not the first such issue:

commit f01db988
Author: Sean Paul <seanpaul@chromium.org>
Date:   Fri Mar 16 12:43:22 2012 -0400

    drm/i915: Add wait_for in init_ring_common

added delay loops to make RING_START and RING_CTL initialization
reliable on the blt ring at boot-up. So I guess it won't hurt if we do
this unconditionally for all force_wake needing gpus.

To avoid copy&pasting of the HAS_FORCE_WAKE check I've added a new
intel_info bit for that.

v2: Fixup missing commas in static struct and properly handling the
error case in init_ring_common, both noticed by Jani Nikula.

Cc: stable@vger.kernel.org
Reported-and-tested-by: NYang Guang <guang.a.yang@intel.com>
Reviewed-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=50522Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

By correctly describing the rinbuffers as being in the GTT domain, it
appears that we are more careful with the management of the CPU cache
upon resume and so prevent some coherency issue when submitting commands
to the GPU later. A secondary effect is that the debug logs are then
consistent with the actual usage (i.e. they no longer describe the
ringbuffers as being in the CPU write domain when we are accessing them
through an wc iomapping.)
Reported-and-tested-by: NDaniel Gnoutcheff <daniel@gnoutcheff.name>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=41092Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@vger.kernel.org
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The previous patch put all the code, and handlers in place. It should
now be safe to enable the parity error interrupt. The parity error must
be unmasked in both the GTIMR, and the CS IMR. Unfortunately, the docs
aren't clear about this; nevertheless it's the truth.
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

When we reset the ring control registers, including the HEAD and TAIL of
the ring, we also need to reset associated state. In this instance, we
were failing to reset the cached value of ring->last_retired_head and so
upon the first request for more space following a resume would
potentially (depending on a narrow race window) believe that the HEAD had
advanced much further than reality.

This is a regression from:

commit a71d8d94
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Wed Feb 15 11:25:36 2012 +0000

    drm/i915: Record the tail at each request and use it to estimate the head
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@vger.kernel.org # 3.4
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Wait request is poorly named IMO. After working with these functions for
some time, I feel it's much clearer to name the functions more
appropriately.

Of course we must update the callers to use the new name as well.

This leaves room within our namespace for a *real* wait request function
at some point.

Note to maintainer: this patch is optional.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

I've flagged this while reviewing the first version and Ken Graunke
fixed it up in v2, but unfortunately Dave Airlie picked up the wrong
version.

Cc: Dave Airlie <airlied@redhat.com>
Cc: Kenneth Graunke <kenneth@whitecape.org>
Cc: stable@kernel.org
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

gpu reset is a very important piece of our infrastructure.
Unfortunately we only really it test by actually hanging the gpu,
which often has bad side-effects for the entire system. And the gpu
hang handling code is one of the rather complicated pieces of code we
have, consisting of
- hang detection
- error capture
- actual gpu reset
- reset of all the gem bookkeeping
- reinitialition of the entire gpu

This patch adds a debugfs to selectively stopping rings by ceasing to
update the hw tail pointer, which will result in the gpu no longer
updating it's head pointer and eventually to the hangcheck firing.
This way we can exercise the gpu hang code under controlled conditions
without a dying gpu taking down the entire systems.

Patch motivated by me forgetting to properly reinitialize ppgtt after
a gpu reset.

Usage:

echo $((1 << $ringnum)) > i915_ring_stop # stops one ring

echo 0xffffffff > i915_ring_stop # stops all, future-proof version

then run whatever testload is desired. i915_ring_stop automatically
resets after a gpu hang is detected to avoid hanging the gpu to fast
and declaring it wedged.

v2: Incorporate feedback from Chris Wilson.

v3: Add the missing cleanup.

v4: Fix up inconsistent size of ring_stop_read vs _write, noticed by
Eugeni Dodonov.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Two things:
- ring->virtual start is an __iomem pointer, treat it accordingly.
- dev_priv->status_page.page_addr is now always a cpu addr, no pointer
  casting needed for that.

Take the opportunity to remove the unnecessary drm indirection when
setting up the ringbuffer iomapping.

v2: Add a compiler barrier before reading the hw status page.
Acked-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We kzalloc dev_priv, and we never use hws_map in intel_ringbuffer.c.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Acked-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This originates from a hack by me to quickly fix a bug in an earlier
patch where we needed control over whether or not waiting on a seqno
actually did any retire list processing. Since the two operations aren't
clearly related, we should pull the parameter out of the wait function,
and make the caller responsible for retiring if the action is desired.

The only function call site which did not get an explicit retire_request call
(on purpose) is i915_gem_inactive_shrink(). That code was already calling
retire_request a second time.

v2: don't modify any behavior excepit i915_gem_inactive_shrink(Daniel)
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We always set it so there's no point in checking. We could
instead add a bit that tells us whether gem is actually
initialized (i.e. either kms or gem_init_ioctl called), but
that's imho not worth it.

So just rip it out.

There's a little change in the wait_ring timeout, but we've never
run with anything else than the 60 second timeout, even on dri1
userspace.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We were attempting to use a per-ring spinlock whilst modifying global
IRQ flags. A recipe for rare missed interrupts.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Acked-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

... and put them to so good use.

Note that there's functional change in vlv clock gating code, we now
no longer spuriously read back the current value of the bit. According
to Bspec the high bits should always read zero, so ORing this in
should have no effect.
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

gen2 hardware has some significant differences from the other interrupt
routines that were glossed over and then forgotten about in the
transition to KMS. Such as

- 16bit IIR
- PendingFlip status bit

This patch reintroduces a handler specifically for gen2 for the purpose
of handling pageflips correctly, simplifying code in the process.

v2: Also fixup ring get/put irq to only access 16bit registers (Daniel)

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=24202
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=41793Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
[danvet: use posting_read16 in intel_ringbuffer.c and kill _driver
from the function names.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Clearing bit 5 of CACHE_MODE_0 is necessary to prevent GPU hangs in
OpenGL programs such as Google MapsGL, Google Earth, and gzdoom when
using separate stencil buffers.  Without it, the GPU tries to use the
LRA eviction policy, which isn't supported.  This was supposed to be off
by default, but seems to be on for many machines.

This cannot be done in gen6_init_clock_gating with most of the other
workaround bits; the render ring needs to exist.  Otherwise, the
register write gets dropped on the floor (one printk will show it
changed, but a second printk immediately following shows the value
reverts to the old one).

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=47535
Cc: stable@vger.kernel.org
Cc: Rob Castle <futuredub@gmail.com>
Cc: Eric Appleman <erappleman@gmail.com>
Cc: aaron667@gmx.net
Cc: Keith Packard <keithp@keithp.com>
Signed-off-by: NKenneth Graunke <kenneth@whitecape.org>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Acked-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NDave Airlie <airlied@redhat.com>

It looks like we also need to flush the render cache when we just
invalidate it. This fixes a regression in i-g-t/gem_tiled_blits on my
i855gm. I guess the render cache there is virtually indexed, so we
need to clean it when changing gtt mappings.

This regression has been introduce in

commit 46f0f8d1
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Wed Apr 18 11:12:11 2012 +0100

    drm/i915: Don't set a MBZ bit in gen2/3 MI_FLUSH
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

On gen2 MI_EXE_FLUSH is actually an AGP flush bit and on gen3 marked as
reserved.  On both it is documented as being must-be-zero. So obey the
documentation, and separate the gen2 flush into its own little routine
and share with gen3.

This means that we can rename the existing render_ring_flush() to
reflect the generation from which it first applies and remove the code
for handling earlier generations from it.

v2: Applies to gen3 as well
v3: Make it compile and improve the commit message.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The (2<<6) virtual memory space selector harks back to gen3 and is
mandatory given our use of GTT space for batchbuffers. On gen4+, use of
the GTT became mandatory and bit6 marked reserved. However the code must
now explicitly set (1<<7), which conveniently is also (2<<6).

To clarify the meaning for future readers, replace the open coded (2<<6)
with MI_BATCH_GTT.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This should contain all the changes which require no thought to make
sparse happy.
Signed-off-by: NBen Widawsky <benjamin.widawsky@intel.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Now that these are properly refactored this additional indirection
doesn't really buy us anything but confusion. Hence inline them.

This duplicates the ironlake gt enable/disable code snippet, but we've
already separate ilk from gen6+ gt irq in i915_irq.c, so I think this
makes more sense.
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We already disallow initialition of gem in this case in the
corresponding ioctl, so don't bother setting up the gem support ring
functions in the legacy dri render ring init.
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

They're indentical, so just kill one. Also give the other a prefix to
distinguish it from the gen6+ functions - this add_request function is
not really generic code.

v2: Fixup commit message as noted by Ben Widawsky.
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Now that we can, we should split them up in a way that makes some
sense and banishes the IS_ checks into init code.
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

HW engineers have fixed this issue for ivb. Again, a nice cleanup
possible thanks to the more flexible ring initialization.
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Now that we have sensibly split up, we can nicely get rid of that ugly
is_gen5 check.
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Inspired by Ben Widawsky's patch for gen6+. Now after restructuring
how we set up the ring vtables and parameters, we can do this right.

This kills the bsd specific get/put_irq functions, they're now the
same.
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The waiter is always the ring itself (otherwise we'd have a decent
snafu in a callsite), so we can unify this easily.

Also give it the usual gen6_ prefix, in case anyone is foolish enough to
implement hw semaphores for gen5.
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

It's not supported, and with the patch to refuse loading on gen6+
without kms enabled, there's also no way we can hit this.
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Just for consistency.
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The same treatment for the bsd ring. Again, this will be split up
further by the irq rework.
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>