This adds support to bind/unbind objects and wires it up. Objects are
only put into the ppgtt when necessary, i.e. at execbuf time.

Objects are still unconditionally put into the global gtt.

v2: Kill the quick hack and explicitly pass cache_level to ppgtt_bind
like for the global gtt function. Noticed by Chris Wilson.
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Tested-by: NChris Wilson <chris@chris-wilson.co.uk>
Tested-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Reviewed-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

On gen5 we also need to correctly set up swizzling in the display
scanout engine, but only there. Consolidate this into the same
function.

This has a small effect on ums setups - the kernel now also sets this
bit in addition to userspace setting it. Given that this code only
runs when userspace either can't (resume, gpu reset) or explicitly
won't(gem_init) touch the hw this shouldn't have an adverse effect.
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We have to do this manually. Somebody had a Great Idea.

I've measured speed-ups just a few percent above the noise level
(below 5% for the best case), but no slowdows. Chris Wilson measured
quite a bit more (10-20% above the usual snb variance) on a more
recent and better tuned version of sna, but also recorded a few
slow-downs on benchmarks know for uglier amounts of snb-induced
variance.

v2: Incorporate Ben Widawsky's preliminary review comments and
elaborate a bit about the performance impact in the changelog.

v3: Add a comment as to why we don't need to check the 3rd memory
channel.

v4: Fixup whitespace.
Acked-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Like for shmem_pwrite_slow. The only difference is that because we
read data, we can leave the fetched cachelines in the cpu: In the case
that the object isn't in the cpu read domain anymore, the clflush for
the next cpu read domain invalidation will simply drop these
cachelines.

slow_shmem_bit17_copy is now ununsed, so kill it.

With this patch tests/gem_mmap_gtt now actually works.

v2: add __ to copy_to_user_swizzled as suggested by Chris Wilson.

v3: Fixup the swizzling logic, it swizzled the wrong pages.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=38115Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

... instead of get_user_pages, because that fails on non page-backed
user addresses like e.g. a gtt mapping of a bo.

To get there essentially copy the vfs read path into pagecache. We
can't call that right away because we have to take care of bit17
swizzling. To not deadlock with our own pagefault handler we need
to completely drop struct_mutex, reducing the atomicty-guarantees
of our userspace abi. Implications for racing with other gem ioctl:

- execbuf, pwrite, pread: Due to -EFAULT fallback to slow paths there's
  already the risk of the pwrite call not being atomic, no degration.
- read/write access to mmaps: already fully racy, no degration.
- set_tiling: Calling set_tiling while reading/writing is already
  pretty much undefined, now it just got a bit worse. set_tiling is
  only called by libdrm on unused/new bos, so no problem.
- set_domain: When changing to the gtt domain while copying (without any
  read/write access, e.g. for synchronization), we might leave unflushed
  data in the cpu caches. The clflush_object at the end of pwrite_slow
  takes care of this problem.
- truncating of purgeable objects: the shmem_read_mapping_page call could
  reinstate backing storage for truncated objects. The check at the end
  of pwrite_slow takes care of this.

v2:
- add missing intel_gtt_chipset_flush
- add __ to copy_from_user_swizzled as suggest by Chris Wilson.

v3: Fixup bit17 swizzling, it swizzled the wrong pages.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The gtt_pwrite slowpath grabs the userspace memory with
get_user_pages. This will not work for non-page backed memory, like a
gtt mmapped gem object. Hence fall throuh to the shmem paths if we hit
-EFAULT in the gtt paths.

Now the shmem paths have exactly the same problem, but this way we
only need to rearrange the code in one write path.

v2: v1 accidentaly falls back to shmem pwrite for phys objects. Fixed.

v3: Make the codeflow around phys_pwrite cleara as suggested by Chris
Wilson.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The original intention of comparing the bo against the mappable GTT
limits was to prevent a subsequent faulting of the bo into the GTT from
clearing the entire GTT in vain. However, that was clearly a cut'n'paste
mistake as a CPU mapping never binds the bo into the aperture. Whilst
there may be some merit to limiting the maximum size of the bo to
something that can be utilized by the GPU, that limit itself does not
belong as a safeguard to mmapping the bo, so remove the check entirely.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

With the fence accounting fixed up in the previous commit not finding
enough fences is a fatal error and userspace bug. Trashing the entire
gtt is not gonna turn up that missing fence, so don't to this by
returning another error thatn ENOSPC.

This has the added benefit that it's easier to distinguish fence
accounting errors from gtt space accounting issues.

TTM serves as precendence for the EDEADLK error code - it returns it
when the reservation code needs resources already blocked by the
current reservation.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

In order to correctly account for reserving space in the GTT and fences
for a batch buffer, we need to independently track whether the fence is
pinned due to a fenced GPU access in the batch or whether the buffer is
pinned in the aperture. Currently we count the fenced as pinned if the
buffer has already been seen in the execbuffer. This leads to a false
accounting of available fence registers, causing frequent mass evictions.
Worse, if coupled with the change to make i915_gem_object_get_fence()
report EDADLK upon fence starvation, the batchbuffer can fail with only
one fence required...

Fixes intel-gpu-tools/tests/gem_fenced_exec_thrash

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=38735Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Tested-by: NPaul Neumann <paul104x@yahoo.de>
[danvet: Resolve the functional conflict with Jesse Barnes sprite
patches, acked by Chris Wilson on irc.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Sometimes it may be the case when we idle the gpu or wait on something
we don't actually want to process the retiring list. This patch allows
callers to choose the behavior.
Reviewed-by: NKeith Packard <keithp@keithp.com>
Reviewed-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

LLC is not SNB/IVB-specific, so we should check for it in a more generic
way.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NKenneth Graunke <kenneth@whitecape.org>
Signed-off-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The waits we do here are generally so short that sleeping is a bad
idea unless we have an IRQ to wake us up.  Improves regression test
performance from 18 minutes to 3.5 minutes on gen7, which is now
consistent with the previous generation.
Signed-off-by: NEric Anholt <eric@anholt.net>
Tested-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Reviewed-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Acked-by: NKenneth Graunke <kenneth@whitecape.org>
Signed-off-by: NKeith Packard <keithp@keithp.com>

As a workaround for IRQ synchronization issues in the gen7 BLT ring,
we want to turn the two wait functions into polling loops.
Signed-off-by: NEric Anholt <eric@anholt.net>
Tested-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Reviewed-by: NEugeni Dodonov <eugeni.dodonov@intel.com>
Acked-by: NKenneth Graunke <kenneth@whitecape.org>
Signed-off-by: NKeith Packard <keithp@keithp.com>

This reverts commit eb1711bb.

It blows up the i915 seqno tracking, resulting in the

	BUG_ON(seqno == 0);

in i915_wait_request() triggering, which will cause lock-ups.

See for example
  https://bugs.launchpad.net/ubuntu/+source/linux/+bug/903010
  https://lkml.org/lkml/2011/12/14/395Reported-requested-and-tested-by: NDirk Hohndel <dirk@hohndel.org>
Reported-by: NRichard Eames <Richard.Eames@flinders.edu.au>
Reported-by: NRocko Requin <rockorequin@hotmail.com>
Acked-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Cc: Dave Airlie <airlied@redhat.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Keith Packard <keithp@keithp.com>
Cc: Eric Anholt <eric@anholt.net>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The recursion loop goes retire_requests->unbind->gpu_idle->retire_reqeusts.

Every time we go through this we need a
- active object that can be retired
- and there are no other references to that object than the one from
  the active list, so that it gets unbound and freed immediately.
Otherwise the recursion stops. So the recursion is only limited by the
number of objects that fit these requirements sitting in the active list
any time retire_request is called.

Issue exercised by tests/gem_unref_active_buffers from i-g-t.

There's been a decent bikeshed discussion whether it wouldn't be
better to pass around a flag, but imo this is o.k. for such a limited
case that only supports a w/a.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=42180Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: NChris Wilson <chris@chris-wilson>
[ickle- we built better bikesheds, but this keeps the rain off for now]
Tested-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

A call to i915_add_request() has been made in function i915_gem_busy_ioctl(). i915_add_request can fail,
so in it's exit path previously allocated memory needs to be freed.
Signed-off-by: NRakib Mullick <rakib.mullick@gmail.com>
Reviewed-by: NKeith Packard <keithp@keithp.com>
Signed-off-by: NKeith Packard <keithp@keithp.com>

I've been seeing memory leaks on my system in the form of large
(300-400MB) GEM objects created by now-dead processes laying around
clogging up memory.  I usually notice when it gets to about 1.2GB of
them.  Hopefully this clears up the issue, but I just found this bug
by inspection.
Signed-off-by: NEric Anholt <eric@anholt.net>
Cc: stable@kernel.org
Signed-off-by: NKeith Packard <keithp@keithp.com>

IVB supports these bits as well.
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NKeith Packard <keithp@keithp.com>

In preparation of to support 32 fences on Ivybdrigde.
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NKeith Packard <keithp@keithp.com>

I've been seeing memory leaks on my system in the form of large
(300-400MB) GEM objects created by now-dead processes laying around
clogging up memory.  I usually notice when it gets to about 1.2GB of
them.  Hopefully this clears up the issue, but I just found this bug
by inspection.
Signed-off-by: NEric Anholt <eric@anholt.net>
Cc: stable@kernel.org
Signed-off-by: NKeith Packard <keithp@keithp.com>

[Description from: Daniel Vetter]
I've just discussed this quickly with Chris on irc and it's probably
best to just kill the list_empty early bailout. gpu_idle isn't a
fastpath, so who cares. One candidate where we emit commands to the ring
without adding anything onto these lists is e.g. pageflip. There are
probably more.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NKeith Packard <keithp@keithp.com>

Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NKeith Packard <keithp@keithp.com>

We currently only round up the userspace size to the next page. We
assume that userspace hasn't made a mistake and requested a zero-length
gem object and all through our internal code we then presume that every
object is backed by at least a single page. Fix that oversight and
report EINVAL back to userspace if they try to create a zero length
object.

[danvet: This fixes tests/gem_bad_length]
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NKeith Packard <keithp@keithp.com>

Use the helper function already employed by the pwrite/pread
functions.
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NKeith Packard <keithp@keithp.com>

Various issues involved with the space character were generating
warnings in the checkpatch.pl file. This patch removes most of those
warnings.
Signed-off-by: NAkshay Joshi <me@akshayjoshi.com>
Signed-off-by: NKeith Packard <keithp@keithp.com>

Signed-off-by: NRob Clark <rob@ti.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Failing to pin a scanout buffer will most likely lead to a black
screen, so if the GPU is wedged, then just let the pin happen and hope
that things work out OK.
Signed-off-by: NKeith Packard <keithp@keithp.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>

Failing to pin a scanout buffer will most likely lead to a black
screen, so if the GPU is wedged, then just let the pin happen and hope
that things work out OK.

v2: Just ignore any error from i915_gem_object_wait_rendering, as
suggested by Chris Wilson
Signed-off-by: NKeith Packard <keithp@keithp.com>

Align unfenced buffers on older hardware to the power-of-two object
size.  The docs suggest that it should be possible to align only to a
power-of-two tile height, but using the already computed fence size is
easier and always correct. We also have to make sure that we unbind
misaligned buffers upon tiling changes.

In order to prevent a repetition of this bug, we change the interface
to the alignment computation routines to force the caller to provide
the requested alignment and size of the GTT binding rather than assume
the current values on the object.
Reported-and-tested-by: NSitosfe Wheeler <sitsofe@yahoo.com>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=36326Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NKeith Packard <keithp@keithp.com>

Provide a parameter to disable hanghcheck. This is useful mostly for
developers trying to debug known problems, and probably should not be
touched by normal users.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NKeith Packard <keithp@keithp.com>

Konstantin Belousov pointed out that 4697995b replaced the generic
i915_driver_irq_*install() functions with chipset specific routines
accessible only through driver->irq_*install(). So update the sanity
check in i915_request_wait() to match.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NKeith Packard <keithp@keithp.com>

The interface to ->truncate_range is changing very slightly: once "tmpfs:
take control of its truncate_range" has been applied, this can be applied.
 For now there is only a slight inefficiency while this remains unapplied,
but it will soon become essential for managing shmem's use of swap.

Change i915_gem_object_truncate() to use shmem_truncate_range() directly:
which should also spare i915 later change if we switch from
inode_operations->truncate_range to file_operations->fallocate.
Signed-off-by: NHugh Dickins <hughd@google.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Keith Packard <keithp@keithp.com>
Cc: Dave Airlie <airlied@redhat.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Soon tmpfs will stop supporting ->readpage and read_cache_page_gfp(): once
"tmpfs: add shmem_read_mapping_page_gfp" has been applied, this patch can
be applied to ease the transition.

Make i915_gem_object_get_pages_gtt() use shmem_read_mapping_page_gfp() in
the one place it's needed; elsewhere use shmem_read_mapping_page(), with
the mapping's gfp_mask properly initialized.

Forget about __GFP_COLD: since tmpfs initializes its pages with memset,
asking for a cold page is counter-productive.

Include linux/shmem_fs.h also in drm_gem.c: with shmem_file_setup() now
declared there too, we shall remove the prototype from linux/mm.h later.
Signed-off-by: NHugh Dickins <hughd@google.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Keith Packard <keithp@keithp.com>
Cc: Dave Airlie <airlied@redhat.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Even if the object is no longer in the GTT domain, there may still be
a user space mapping which needs to be released.

Without this fix, render-based text (mostly in firefox) would
occasionally get corrupted when the system was under load.
Signed-off-by: NKeith Packard <keithp@keithp.com>

This reverts commit 4a684a41.
Userland has always been required to set the object's domain to GTT
before using it through a GTT mapping, it's not something that the
kernel is supposed to enforce.  (The pagefault support is so that we
can handle multiple mappings without userland having to pin across
them, not so that userland can use GTT after GPU domains without
telling the kernel).

Fixes 19.2% +/- 0.8% (n=6) performance regression in cairo-gl
firefox-talos-gfx on my T420 latop.
Signed-off-by: NKeith Packard <keithp@keithp.com>

It seems to me that we are leaking 'user_pages' in
drivers/gpu/drm/i915/i915_gem.c::i915_gem_shmem_pread_slow() if
read_cache_page_gfp() fails.
Signed-off-by: NJesper Juhl <jj@chaosbits.net>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Improves full-screen openarena on my laptop 20.3% +/- 4.0% (n=3)
Improves 800x600 nexuiz on my laptop 12.3% +/- 0.1% (n=3)

We have more room to improve with doing LLC caching for display using
GFDT, and in doing LLC+MLC caching, but this was an easy performance
win and incremental improvement toward those two.
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The simplest and common method for ensuring scanout coherency on all
chipsets is to mark the scanout buffers as uncached (and for
userspace to remember to flush the render cache every so often).

We can improve upon this for later generations by marking scanout
objects as GFDT and only flush those cachelines when required. However,
we start simple.

[v2: Move the set to uncached above the clflush.  Otherwise, we'd skip
the clflush and try to scan out data that was still sitting in the
cache.]
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We need to perform a few operations in order to move the object into the
display plane (where it can be accessed coherently by the display
engine) that are important for future safety to forbid whilst pinned. As a
result, we want to need to perform some of the operations before pinning,
but some are required once we have been bound into the GTT. So combine
the pinning performed by all the callers with set_to_display_plane(), so
this complication is contained within the single function.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

[anholt v2: Don't forget that when going from cached to uncached, we
haven't been tracking the write domain from the CPU perspective, since
we haven't needed it for GPU coherency.]

[ickle v3: We also need to make sure we relinquish any fences on older
chipsets and clear the GTT for sane domain tracking.]
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>