Based on earlier work by Chris Wilson <chris@chris-wilson.co.uk>

CC: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NJani Nikula <jani.nikula@intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Some BIOSes may forcibly suspend RC6 during their operation which
trigger a warning as we find the hardware in a perplexing state upon
first use. So far that appears to be the worst symptom as fortuituously
we use the same values as the BIOS for programming the FORCEWAKE register.
Reported-by: NOleksij Rempel <bug-track@fisher-privat.net>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

There is a special mechanism for communicating with the PCU already
being used for the ring frequency stuff. As we'll be needing this for
other commands, extract it now to make future code less error prone and
the current code more reusable.

I'm not entirely sure if this code matches 1:1 with the previous code
behaviorally. Functionally however, it should be the same.

CC: Jesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
[danvet: Fixup compile fail reported by Wu Fengguang.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Note that just because we have n == MAX elements left, does not imply
that there are only MAX elements left in the scatterlist and so we may
not be on the last chain, and the nth element may in fact be a chain ptr.

This is exercised by the improved hangman tests and the gem_exec_big
test in i-g-t.

This regression has been introduced in

commit 9da3da66
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Fri Jun 1 15:20:22 2012 +0100

   drm/i915: Replace the array of pages with a scatterlist

v2: KISS, replace the direct lookup with a for_each_sg() [danvet]
v3: Try to be clever again.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The intention was to allow the caller to avoid a failure to queue a
request having already written commands to the ring. However, this is a
moot point as the i915_add_request() can fail for other reasons than a
mere allocation failure and those failure cases are more likely than
ENOMEM. So the overlay code already had to handle i915_add_request()
failures, and due to

commit 3bb73aba
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Fri Jul 20 12:40:59 2012 +0100

    drm/i915: Allow late allocation of request for i915_add_request()

the error handling code in intel_overlay.c was subject to causing
double-frees, as found by coverity.

Rather than further complicate i915_add_request() and callers, realise
the battle is lost and adapt intel_overlay.c to take advantage of the
late allocation of requests.

v2: Handle callers passing in a NULL seqno.
v3: Ditto. This time for sure.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NJani Nikula <jani.nikula@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Problems with the previous code:
  - HDMI just uses WRPLL1 for everything, so dual head cases might not
    work sometimes.
  - At encoder->mode_set we just write the PLL register without doing
    any kind of check (e.g., check if the PLL is already being used).
  - There is no way to fail and return error codes at
    encoder->mode_set.
  - We write to PORT_CLK_SEL at mode_set and we never disable it.
  - Machines hang due to wrong clock enable/disable sequence.

So here we rewrite the code, making it a little more like the
pre-Haswell PLL mode set code:
  - Check PLL availability at ironlake_crtc_mode_set.
  - Try to use both WRPLLs.
  - Check if PLLs are used before actually trying to use them, and
    properly fail with error messages.
  - Enable/disable PORT_CLK_SEL at the right place.
  - Add some WARNs to check for bugs.

The next improvement will be to try to reuse PLLs if the timings
match, but this is content for another patch and it's already
documented with a TODO comment.
Signed-off-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

round_jiffies() aligns the wakeup time to the nearest second in order to
batch wakeups and reduce system load, which is useful for unimportant
coarse timers like our hangcheck.

v2: round_jiffies_relative() returns the relative jiffie value, whereas
we need the absolute value for the timer.
Suggested-by: NArjan van de Ven <arjan@linux.intel.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Reviewed-by: NJani Nikula <jani.nikula@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

By providing a callback for when we need to bind the pages, and then
release them again later, we can shorten the amount of time we hold the
foreign pages mapped and pinned, and importantly the dmabuf objects then
behave as any other normal object with respect to the shrinker and
memory management.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Magic numbers are bad mmmkay. In this case in particular the value is
especially weird because the docs say multiple things. We'll need this
value for sysfs, so extracting it is useful for that as well.
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Rather than have multiple data structures for describing our page layout
in conjunction with the array of pages, we can migrate all users over to
a scatterlist.

One major advantage, other than unifying the page tracking structures,
this offers is that we replace the vmalloc'ed array (which can be up to
a megabyte in size) with a chain of individual pages which helps reduce
memory pressure.

The disadvantage is that we then do not have a simple array to iterate,
or to access randomly. The common case for this is in the relocation
processing, which will typically fit within a single scatterlist page
and so be almost the same cost as the simple array. For iterating over
the array, the extra function call could be optimised away, but in
reality is an insignificant cost of either binding the pages, or
performing the pwrite/pread.

v2: Fix drm_clflush_sg() to not invoke wbinvd as well! And fix the
trivial compile error from rebasing.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We need to refcount our pages in order to prevent reaping them at
inopportune times, such as when they currently vmapped or exported to
another driver. However, we also wish to keep the lazy deallocation of
our pages so we need to take a pin/unpinned approach rather than a
simple refcount.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

In order to specialise functions depending upon the type of object, we
can attach vfuncs to each object via a new ->ops pointer.

For instance, this will be used in future patches to only bind pages from
a dma-buf for the duration that the object is used by the GPU - and so
prevent them from pinning those pages for the entire of the object.

v2: Bonus comments.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

... instead of resetting a few things and hoping that this will work
out.

To properly disable the output pipelines at the initial modeset after
resume or boot up we need to have an accurate picture of which outputs
are enabled and connected to which crtcs. Otherwise we risk disabling
things at the wrong time, which can lead to hangs (or at least royally
confused panels), both requiring a walk to the reset button to fix.

Hence read out the hw state with the freshly introduce get_hw_state
functions and then sanitize it afterwards.

For a full modeset readout (which would allow us to avoid the initial
modeset at boot up) a few things are still missing:
- Reading out the mode from the pipe, especially the dotclock
  computation is quite some fun.
- Reading out the parameters for the stolen memory framebuffer and
  wrapping it up.
- Reading out the pch pll connections - luckily the disable code
  simply bails out if the crtc doesn't have a pch pll attached (even
  for configurations that would need one).

This patch here turned up tons of smelly stuff around resume: We
restore tons of register in seemingly random way (well, not quite, but
we're not too careful either), which leaves the hw in a rather
ill-defined state: E.g. the port registers are sometimes
unconditionally restore (lvds, crt), leaving us with an active
encoder/connector but no active pipe connected to it. Luckily the hw
state sanitizer detects this madness and fixes things up a bit.

v2: When checking whether an encoder with active connectors has a crtc
wire up to it, check for both the crtc _and_ it's active state.

v3:
- Extract intel_sanitize_encoder.
- Manually disable active encoders without an active pipe.

v4: Correclty fix up the pipe<->plane mapping on machines where we
switch pipes/planes. Noticed by Chris Wilson, who also provided the
fixup.

v5: Spelling fix in a comment, noticed by Paulo Zanoni
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Because that's what we're essentially calling. This is the first step
in untangling the crtc_helper induced dpms handling mess we have - at
the crtc level we only have 2 states and the magic is just in
selecting which one (and atm there isn't even much magic, but on
recent platforms where not even the crt output has more than 2 states
we could do better).
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Like with the equivalent change for gen6+ rps state, this helps in
clarifying the code (and in fixing a few places that have fallen through
the cracks in the locking review).
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Using the extracted INSTDONE reading, and our new register definitions,
update our hangcheck detection and error collection to use it. This
primarily means changing == to memcmp, and changing = to memcpy.
Hopefully this will give more info on error dump, and provide more
accurate hangcheck detection (both are actually TBD).

Also, remove the reading of instdone1 from the ring error collection
function, and just crap everything in capture_error_state (that could be
split into a separate patch if it wasn't so trivial).

v2: Now assuming i915_get_extra_instdone does the memset we can clean up the
code a bit (Jani)

v3: use ARRAY_SIZE as requested earlier by Jani (didn't change sizeof)
Updated commit msg

Cc: Jani Nikula <jani.nikula@intel.com>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NJani Nikula <jani.nikula@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

As we wish to create specialised object constructions in the near
future that share the same basic GEM object struct, export the default
initializer.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Avoid stalling and waiting for the GPU by checking to see if there is
sufficient inactive space in the aperture for us to bind the buffer
prior to writing through the GTT. If there is inadequate space we will
have to stall waiting for the GPU, and incur overheads moving objects
about. Instead, only incur the clflush overhead on the target object by
writing through shmem.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

ERR_INT can generate interrupts. However since most of the conditions seem
quite fatal the patch opts to simply report it in error state instead of
adding more complexity to the interrupt handler for little gain (the
bits are sticky anyway).
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Tested-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Reviewed-by: NAntti Koskipaa <antti.koskipaa@linux.intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

When dealing with a working set larger than the GATT, or even the
mappable aperture when touching through the GTT, we end up with evicting
objects only to rebind them at a new offset again later. Moving an
object into and out of the GTT requires clflushing the pages, thus
causing a double-clflush penalty for rebinding.

To avoid having to clflush on rebinding, we can track the pages as they
are evicted from the GTT and only relinquish those pages on memory
pressure.

As usual, if it were not for the handling of out-of-memory condition and
having to manually shrink our own bo caches, it would be a net reduction
of code. Alas.

Note: The patch also contains a few changes to the last-hope
evict_everything logic in i916_gem_execbuffer.c - we no longer try to
only evict the purgeable stuff in a first try (since that's superflous
and only helps in OOM corner-cases, not fragmented-gtt trashing
situations).

Also, the extraction of the get_pages retry loop from bind_to_gtt (and
other callsites) to get_pages should imo have been a separate patch.

v2: Ditch the newly added put_pages (for unbound objects only) in
i915_gem_reset. A quick irc discussion hasn't revealed any important
reason for this, so if we need this, I'd like to have a git blame'able
explanation for it.

v3: Undo the s/drm_malloc_ab/kmalloc/ in get_pages that Chris noticed.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
[danvet: Split out code movements and rant a bit in the commit message
with a few Notes. Done v2]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

... and move a few others only used by i915_dma.c into the dri1
dungeon.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

It's only ever a pointer to the global mchdev_lock, and we don't use
it at all.
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This way it's easier so see what belongs together, and what is used
by the ilk ips code. Also add some comments that explain the locking.

Note that (cur|min|max)_delay need to be duplicated, because
they're also used by the ips code.

v2: Missed one place that the dev_priv->ips change caught ...
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NDamien Lespiau <damien.lespiau@intel.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Handy for lazy people like me, or when people forget to add the output
of lspci -nn.

v2: Chris Wilson noticed that we have this duplicated already in the
i915_capabilites debugfs file. But there \n as separator looks better,
which would be a bit verbose in dmesg. Abuse the preprocessor to
extract this all.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We believe to have squashed all issues around the gen6+ rps interrupt
generation and why the gpu sometimes got stuck. With that cleared up,
there's no user left for the sanitize_pm infrastructure, so let's just
rip it out.

Note that 'intel_reg_write 0xa014 0x13070000' is the w/a if we find
ourselves stuck again.
Acked-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

By selecting the cache level (essentially whether or not the CPU snoops
any updates to the bo, and on more recent machines whether it resides
inside the CPU's last-level-cache) a userspace driver is able to then
manage all of its memory within buffer objects, if it so desires. This
enables the userspace driver to accelerate uploads and more importantly
downloads from the GPU and to able to mix CPU and GPU rendering/activity
efficiently.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
[danvet: Added code comment about where we plan to stuff platform
specific cacheing control bits in the ioctl struct.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Several functions of the GPU have the restriction that differing memory
domains cannot be placed next to each other (as the GPU may prefetch
beyond the end of one domain and hang as it crosses into the other
domain). We use the facility of the drm_mm to mark ranges with a
particular color that corresponds to the cache attributes of those pages
in order to prevent allocating adjacent blocks of differing memory
types.

v2: Rebase ontop of drm_mm coloring v2.
v3: Fix rebinding existing gtt_space and add a verification routine.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Originally I had a macro specifically for DPF support, and Daniel, with
good reason asked me to change it to this. It's not the way I would have
gone (and indeed I didn't), but for now there is no distinction as all
platforms with L3 also have DPF.

Note: The good reasons are that dpf is a l3$ feature (at least on
currrent hw), hence I don't expect one to go without the other.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
[danvet: added note]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

As suggested by Daniel, rip out the independent timers for device and
crtc busyness and integrate the manual powermanagement of the display
engine into the GEM core and its request tracking. The benefits are that
the code is a lot smaller, fewer moving parts and should fit more neatly
into the overall activity tracking of the driver.

v2: Complete overhaul and removal of the racy timers and workers.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

By moving the function to intel_ringbuffer and currying the appropriate
parameter, hopefully we make the callsites easier to read and
understand.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This is now handled by a global flag to ensure we emit a flush before
the next serialisation point (if we failed to queue one previously).
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

As we guarantee to emit a flush before emitting the breadcrumb or
the next batchbuffer, there is no further need for the flushing list.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

As we always flush the GPU cache prior to emitting the breadcrumb, we no
longer have to worry about the deferred flush causing the
pending_gpu_write to be delayed. So we can instead utilize the known
last_write_seqno to hopefully minimise the wait times.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Request preallocation was added to i915_add_request() in order to
support the overlay. However, not all users care and can quite happily
ignore the failure to allocate the request as they will simply repeat
the request in the future.

By pushing the allocation down into i915_add_request(), we can then
remove some rather ugly error handling in the callers.

v2: Nullify request->file_priv otherwise we chase a garbage pointer
when retiring requests.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The interface's immediate purpose is to do synchronous timestamp queries
as required by GL_TIMESTAMP. The GPU has a register for reading the
timestamp but because that would normally require root access through
libpciaccess, the IOCTL can provide this service instead.

Currently the implementation whitelists only the render ring timestamp
register, because that is the only thing we need to expose at this time.

v2: make size implicit based on the register offset
Add a generation check
Reviewed-by: NEric Anholt <eric@anholt.net>
Cc: Jacek Lawrynowicz <jacek.lawrynowicz@intel.com>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
[danvet: fixup the ioctl numerb:]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Having had to dive into the bspec to understand what each stage of the
workaround meant, and how that the ring broadcasting IDLE corresponded
with the GT powering down the ring (i.e. rc6) add comments to aide
the next reader.

And since the register "is used to control all aspects of PSMI and power
saving functions" that makes it quite interesting to inspect with
regards to RC6 hangs, so add it to the error-state.

v2: Rediscover the piece of magic, set the RNCID to 0 before waiting for
the ring to wake up.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We already have this pattern at quite a few places, and moving part of
the modeset helper stuff into the driver will add more.

v2: Don't clobber the crtc struct name with the macro parameter ...

v3: Convert two more places noticed by Paulo Zanoni.
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

So don't return -EAGAIN, even in the case of a gpu hang. Remap it to
-EIO instead. Note that this isn't really an issue with
interruptability, but more that we have quite a few codepaths (mostly
around kms stuff) that simply can't handle any errors and hence not
even -EAGAIN. Instead of adding proper failure paths so that we could
restart these ioctls we've opted for the cheap way out of sleeping
non-interruptibly.  Which works everywhere but when the gpu dies,
which this patch fixes.

So essentially interruptible == false means 'wait for the gpu or die
trying'.'

This patch is a bit ugly because intel_ring_begin is all non-interruptible
and hence only returns -EIO. But as the comment in there says,
auditing all the callsites would be a pain.

To avoid duplicating code, reuse i915_gem_check_wedge in __wait_seqno
and intel_wait_ring_buffer. Also use the opportunity to clarify the
different cases in i915_gem_check_wedge a bit with comments.

v2: Don't access dev_priv->mm.interruptible from check_wedge - we
might not hold dev->struct_mutex, making this racy. Instead pass
interruptible in as a parameter. I've noticed this because I've hit a
BUG_ON(!mutex_is_locked) at the top of check_wedge. This has been
added in

commit b4aca010
Author: Ben Widawsky <ben@bwidawsk.net>
Date:   Wed Apr 25 20:50:12 2012 -0700

    drm/i915: extract some common olr+wedge code

although that commit is missing any justification for this. I guess
it's just copy&paste, because the same commit add the same BUG_ON
check to check_olr, where it indeed makes sense.

But in check_wedge everything we access is protected by other means,
so this is superflous. And because it now gets in the way (we add a
new caller in __wait_seqno, which can be called without
dev->struct_mutext) let's just remove it.

v3: Group all the i915_gem_check_wedge refactoring into this patch, so
that this patch here is all about not returning -EAGAIN to callsites
that can't handle syscall restarting.

v4: Add clarification what interuptible == fales means in our code,
requested by Ben Widawsky.

v5: Fix EAGAIN mispell noticed by Chris Wilson.
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Tested-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>