In order to reduce the penalty of fallbacks under memory pressure and to
avoid a potential immediate ping-pong of evicting a mmaped buffer, we
move the object to the tail of the inactive list when a page is freshly
faulted or the object is moved into the CPU domain.

We choose not to protect the CPU objects from casual eviction,
preferring to keep the GPU active for as long as possible.

v2: Daniel Vetter found a bug where I forgot that pinned objects are
kept off the inactive list.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

The eviction code is the gnarly underbelly of memory management, and is
clearer if kept separated from the normal domain management in GEM.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

This will be used by the eviction logic to maintain fairness between the
rings.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

This does two little changes:

- Add an alignment parameter for evict_something. It's not really great to
  whack a carefully sized hole into the gtt with the wrong alignment.
  Especially since the fallback path is a full evict.

- With the inactive scan stuff we need to evict more that one object, so
  move the unbind call into the helper function that scans for the object
  to be evicted, too.  And adjust its name.

No functional changes in this patch, just preparation.
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

In order to properly track bound objects, they need to exist on one of
the inactive/active lists or be pinned. As this is a requirement, do the
work inside i915_gem_bind_to_gtt() rather than dotted around the
callsites.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

This debugging trace was useful for finding the fbcon regression on
i965, and it may prove useful again in future.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

Incorporates a similar patch by Daniel Vetter, the alteration being to
report the current busy state after retiring.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NEric Anholt <eric@anholt.net>

This avoids the excess flush and requests on idle rings (and spamming
the debug log ;-)
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

This is required should we ever attempt to use an io-mapping where
KM_USER0 is verboten, such as inside an IRQ context.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Eric Anholt <eric@anholt.net>
Signed-off-by: NDave Airlie <airlied@redhat.com>

When creating an object, we create the handle by which it is known to
the process and which own the reference to the object. That reference to
the new handle is what we want to transfer to the process, not the lost
reference to the object; so free the local object reference *not* the
process's handle reference.

This brings i915_gem_object_create_ioctl() into line with
drm_gem_open_ioctl()
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

If we fail to flush outstanding GPU writes but return the memory to the
system, we risk corrupting memory should the GPU recovery and complete
those writes. On the other hand, if we bail early and free the object
then we have a definite use-after-free and real memory corruption.
Choose the lesser of two evils, since in order to recover from the hung
GPU we need to completely reset it, those pending writes should
never happen.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

If during the freeing of an object the unbind is interrupted by a system
call, which is quite possible if we have outstanding GPU writes that
must be flushed, the unbind is silently aborted. This still leaves the
AGP region and backing pages allocated, and perhaps more importantly,
the object remains upon the various lists exposing us to memory
corruption.

I think this is the cause behind the use-after-free, such as

  Bug 15664 - Graphics hang and kernel backtrace when starting Azureus
              with Compiz enabled
  https://bugzilla.kernel.org/show_bug.cgi?id=15664

v2: Daniel Vetter reminded me that kernel space programming is never easy.
We cannot simply spin to clear the pending signal and so must deferred
the freeing of the object until later.
v3: Run from the top level retire requests.
v4: Tested with P(return -ERESTARTSYS)=.5 from i915_gem_do_wait_request()
v5: Rebase against Eric's for-linus tree.
v6: Refactor, split and add a comment about avoiding unbounded recursion.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Daniel Vetter <daniel@ffwll.ch>
Signed-off-by: NEric Anholt <eric@anholt.net>

Combine the iteration over active render rings into a common function.
This is in preparation for reusing the idle function to also retire
deferred free requests.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

Simple fix for error propagation along the old UMS path.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

Oops, when merging the extra details following an OOM, I missed that
driver_private is now NULL and the correct way to convert from the
drm_gem_object into the drm_i915_gem_object is to use to_intel_bo().

BUG: unable to handle kernel NULL pointer dereference at 00000069
IP: [<c11a4a02>] i915_gem_do_execbuffer+0x71f/0xbb6
*pde = 00000000
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/virtual/vc/vcsa3/uevent

Pid: 10993, comm: X Not tainted 2.6.35-rc2+ #67 /
EIP: 0060:[<c11a4a02>] EFLAGS: 00213202 CPU: 0
EIP is at i915_gem_do_execbuffer+0x71f/0xbb6
EAX: f647e8a8 EBX: 00000000 ECX: 00000003 EDX: 00000000
ESI: 00424000 EDI: 00000000 EBP: f6508e48 ESP: f6508dd4
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process X (pid: 10993, ti=f6508000 task=f6432880 task.ti=f6508000)
Stack:
 f6508de0 f7130000 00000001 00000000 00000000 f647e8a8 00000000 f64f8480
<0> f7974414 00000000 00000006 00000000 00000000 f6578000 00000008 00000006
<0> f6797880 00400000 00000000 ffffffe4 f7974400 000000d0 000000d0 000001c0
Call Trace:
 [<c11a4f3a>] ? i915_gem_execbuffer2+0xa1/0xe7
 [<c118ab96>] ? drm_ioctl+0x22c/0x2fa
 [<c11a4e99>] ? i915_gem_execbuffer2+0x0/0xe7
 [<c107e88c>] ? do_sync_read+0x8f/0xca
 [<c1088cbd>] ? vfs_ioctl+0x2c/0x96
 [<c118a96a>] ? drm_ioctl+0x0/0x2fa
 [<c10891f4>] ? do_vfs_ioctl+0x429/0x45a
 [<c107e5c9>] ? fsnotify_access+0x54/0x5f
 [<c107ee1c>] ? vfs_read+0x9a/0xae
 [<c1089258>] ? sys_ioctl+0x33/0x4d
 [<c1002610>] ? sysenter_do_call+0x12/0x26
Code: d0 89 4d c4 31 c9 89 45 d8 eb 44 8b 45 cc 8b 14 88 8b 42 50 89 45
bc 8b 45 a0 8b 52 38 89 55 d0 31 d2 f6 40 20 01 74 0d 8b 55 bc <f6> 42
69 30 0f 95 c2 0f b6 d2 8b 45 d0 c7 45 d4 00 00 00 00 89
EIP: [<c11a4a02>] i915_gem_do_execbuffer+0x71f/0xbb6 SS:ESP 0068:f6508dd4
CR2: 0000000000000069
---[ end trace 3f1d514b34d39381 ]---
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

A lot of 945GMs have had stability issues for a long time, this manifested as X hangs, blitter engine hangs, and lots of crashes.

one such report is at:
https://bugs.freedesktop.org/show_bug.cgi?id=20560

along with numerous distro bugzillas.

This only took a week of digging and hair ripping to figure out.

Tracked down and tested on a 945GM Lenovo T60,
previously running
x11perf -copypixwin500
or
x11perf -copywinpix500
repeatedly would cause the GPU to wedge within 4 or 5 tries, with random busy bits set.

After this patch no hangs were observed.

cc: stable@kernel.org
Signed-off-by: NDave Airlie <airlied@redhat.com>

The current shrinker implementation requires the registered callback
to have global state to work from. This makes it difficult to shrink
caches that are not global (e.g. per-filesystem caches). Pass the shrinker
structure to the callback so that users can embed the shrinker structure
in the context the shrinker needs to operate on and get back to it in the
callback via container_of().
Signed-off-by: NDave Chinner <dchinner@redhat.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

The hibernate issues that got fixed in commit 985b823b ("drm/i915:
fix hibernation since i915 self-reclaim fixes") turn out to have been
incomplete.  Vefa Bicakci tested lots of hibernate cycles, and without
the __GFP_RECLAIMABLE flag the system eventually fails to resume.

With the flag added, Vefa can apparently hibernate forever (or until he
gets bored running his automated scripts, whichever comes first).

The reclaimable flag was there originally, and was one of the flags that
were dropped (unintentionally) by commit 4bdadb97 ("drm/i915:
Selectively enable self-reclaim") that introduced all these problems,
but I didn't want to just blindly add back all the flags in commit
985b823b, and it looked like __GFP_RECLAIM wasn't necessary.  It
clearly was.

I still suspect that there is some subtle reason we're missing that
causes the problems, but __GFP_RECLAIMABLE is certainly not wrong to use
in this context, and is what the code historically used.  And we have no
idea what the causes the corruption without it.
Reported-and-tested-by: NM. Vefa Bicakci <bicave@superonline.com>
Cc: Dave Airlie <airlied@gmail.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
Cc: stable@kernel.org
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Only ever assigned, never used.
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
[glisse: I will re-add if needed for range-restricted allocations]
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Oops, when merging the extra details following an OOM, I missed that
driver_private is now NULL and the correct way to convert from the
drm_gem_object into the drm_i915_gem_object is to use to_intel_bo().

BUG: unable to handle kernel NULL pointer dereference at 00000069
IP: [<c11a4a02>] i915_gem_do_execbuffer+0x71f/0xbb6
*pde = 00000000
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/virtual/vc/vcsa3/uevent

Pid: 10993, comm: X Not tainted 2.6.35-rc2+ #67 /
EIP: 0060:[<c11a4a02>] EFLAGS: 00213202 CPU: 0
EIP is at i915_gem_do_execbuffer+0x71f/0xbb6
EAX: f647e8a8 EBX: 00000000 ECX: 00000003 EDX: 00000000
ESI: 00424000 EDI: 00000000 EBP: f6508e48 ESP: f6508dd4
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process X (pid: 10993, ti=f6508000 task=f6432880 task.ti=f6508000)
Stack:
 f6508de0 f7130000 00000001 00000000 00000000 f647e8a8 00000000 f64f8480
<0> f7974414 00000000 00000006 00000000 00000000 f6578000 00000008 00000006
<0> f6797880 00400000 00000000 ffffffe4 f7974400 000000d0 000000d0 000001c0
Call Trace:
 [<c11a4f3a>] ? i915_gem_execbuffer2+0xa1/0xe7
 [<c118ab96>] ? drm_ioctl+0x22c/0x2fa
 [<c11a4e99>] ? i915_gem_execbuffer2+0x0/0xe7
 [<c107e88c>] ? do_sync_read+0x8f/0xca
 [<c1088cbd>] ? vfs_ioctl+0x2c/0x96
 [<c118a96a>] ? drm_ioctl+0x0/0x2fa
 [<c10891f4>] ? do_vfs_ioctl+0x429/0x45a
 [<c107e5c9>] ? fsnotify_access+0x54/0x5f
 [<c107ee1c>] ? vfs_read+0x9a/0xae
 [<c1089258>] ? sys_ioctl+0x33/0x4d
 [<c1002610>] ? sysenter_do_call+0x12/0x26
Code: d0 89 4d c4 31 c9 89 45 d8 eb 44 8b 45 cc 8b 14 88 8b 42 50 89 45
bc 8b 45 a0 8b 52 38 89 55 d0 31 d2 f6 40 20 01 74 0d 8b 55 bc <f6> 42
69 30 0f 95 c2 0f b6 d2 8b 45 d0 c7 45 d4 00 00 00 00 89
EIP: [<c11a4a02>] i915_gem_do_execbuffer+0x71f/0xbb6 SS:ESP 0068:f6508dd4
CR2: 0000000000000069
---[ end trace 3f1d514b34d39381 ]---
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

Since commit 4bdadb97 ("drm/i915:
Selectively enable self-reclaim"), we've been passing GFP_MOVABLE to the
i915 page allocator where we weren't before due to some over-eager
removal of the page mapping gfp_flags games the code used to play.

This caused hibernate on Intel hardware to result in a lot of memory
corruptions on resume.  See for example

  http://bugzilla.kernel.org/show_bug.cgi?id=13811

Reported-by: Evengi Golov (in bugzilla)
Signed-off-by: NDave Airlie <airlied@redhat.com>
Tested-by: NM. Vefa Bicakci <bicave@superonline.com>
Cc: stable@kernel.org
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Since we now get_user_pages() outside of the mutex prior to performing
the copy, we kmap() the page inside the copy routine and so need to
perform an ordinary memcpy() and not copy_from_user().
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

As we do not have a requirement to be atomic and avoid sleeping whilst
performing the slow copy for shmem based pread and pwrite, we can use
kmap instead, thus simplifying the code.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

We can avoid an early clflush when pwriting if we use the current CPU
write domain rather than moving the object to the GTT domain for the
purposes of the pwrite. This has the advantage of not flushing the
presumably hot data that we want to upload into the bo, and of ascribing
the clflush to the execution when profiling.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

The callers expect us to cleanup any partially initialised structures
before reporting the error.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

If the object is bigger than the entire aperture, reject it early
before evicting everything in a vain attempt to find space.

v2: Use E2BIG as suggested by Owain G. Ainsworth.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org
Signed-off-by: NEric Anholt <eric@anholt.net>

This particular warning is harmless as we emit during the normal
pinning process where the batch buffer requires more fences than is
available without eviction. Only if we fail to evict enough fences does
this become a problem, so include the requested number of fences in the
ultimate *error* message.

v2: Remember to compile test even trial patches to remove warnings.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

Whilst pinning the buffer, check that that its current alignment
matches the requested alignment. If it does not, rebind.

This should clear up any final render errors whilst resuming,
for reference:

  Bug 27070 - [i915] Page table errors with empty ringbuffer
  https://bugs.freedesktop.org/show_bug.cgi?id=27070

  Bug 15502 -  render error detected, EIR: 0x00000010
  https://bugzilla.kernel.org/show_bug.cgi?id=15502

  Bug 13844 -  i915 error: "render error detected"
  https://bugzilla.kernel.org/show_bug.cgi?id=13844Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org
Signed-off-by: NEric Anholt <eric@anholt.net>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NEric Anholt <eric@anholt.net>

Nesting domain changes will cause confusion when trying to interpret the
tracepoints describing the sequence of changes for the object, as well
as obscuring the order of operations for the reader of the code.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

This saves a whooping 7 dwords. Zero functional changes. Because
some of the refcounts are rather tightly calculated, I've put
BUG_ONs in the code to check for overflows.
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NEric Anholt <eric@anholt.net>

The BSD (bit stream decoder) ring is used for accessing the BSD engine
which decodes video bitstream for H.264 and VC1 on G45+.  It is
asynchronous with the render ring and has access to separate parts of
the GPU from it, though the render cache is coherent between the two.
Signed-off-by: NZou Nan hai <nanhai.zou@intel.com>
Signed-off-by: NXiang Hai hao <haihao.xiang@intel.com>
Signed-off-by: NEric Anholt <eric@anholt.net>

The active list and request list move into the ringbuffer structure,
so each can track its active objects in the order they are in that
ring.  The flushing list does not, as it doesn't matter which ring
caused data to end up in the render cache.  Objects gain a pointer to
the ring they are active on (if any).
Signed-off-by: NZou Nan hai <nanhai.zou@intel.com>
Signed-off-by: NXiang Hai hao <haihao.xiang@intel.com>
Signed-off-by: NEric Anholt <eric@anholt.net>

Introduces a more complete intel_ring_buffer structure with callbacks
for setup and management of a particular ringbuffer, and converts the
render ring buffer consumers to use it.
Signed-off-by: NZou Nan hai <nanhai.zou@intel.com>
Signed-off-by: NXiang Hai hao <haihao.xiang@intel.com>
[anholt: Fixed up whitespace fail and rebased against prep patches]
Signed-off-by: NEric Anholt <eric@anholt.net>

With the advent of the BSD ring, be clear about which ring this is.
The docs are pretty consistent with calling this the Render engine at
this point.

This is preparation for supporting multiple ringbuffers on Ironlake.
The non-copy-and-paste changes are:
- de-staticing functions
- I915_GEM_GPU_DOMAINS moving to i915_drv.h to be used by both files.
- i915_gem_add_request had only half its implementation
  copy-and-pasted out of the middle of it.

This lru tracks fences, not objects, so move it to where it belongs.
As a side effect, this nicely shrinks drm_i915_gem_object by two
pointers.
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NEric Anholt <eric@anholt.net>

By idling the GPU and discarding everything we can when under extreme
memory pressure, the number of OOM-killer events is dramatically
reduced. For instance, this makes it possible to run
firefox-planet-gnome.trace again on my swapless 512MiB i915.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

My PIPE_CONTROL fix (just sent via Eric's tree) was buggy; I was
testing a whole set of patches together and missed a conversion to the
new HAS_PIPE_CONTROL macro, which will cause breakage on non-Ironlake
965 class chips.  Fortunately, the fix is trivial and has been tested.

Be sure to use the HAS_PIPE_CONTROL macro in i915_get_gem_seqno, or
we'll end up reading the wrong graphics memory, likely causing hangs,
crashes, or worse.
Reported-by: NZdenek Kabelac <zdenek.kabelac@gmail.com>
Reported-by: NToralf Förster <toralf.foerster@gmx.de>
Tested-by: NToralf Förster <toralf.foerster@gmx.de>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>