Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

More precisely: For those that _need_ to be mappable. Also add two
BUG_ONs in fault and pin to check the consistency of the mappable
flag.

Changes in v2:
- Add tracking of gtt mappable space (to notice mappable/unmappable
  balancing issues).
- Improve the mappable working set tracking by tracking fault and pin
  separately.
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

This way we can make some more educated guesses as to why exactly
we can't use 2G apertures to their full potential ;)
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

At least the part that's currently enabled by the BIOS.
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

In i915_gem_object_pin obviously unbind only if mappable is true.

This is the last part to enable gtt_mappable_end != gtt_size, which
the next patch will do.

v2: Fences on g33/pineview only work in the mappable part of the
gtt.
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Like before add a parameter mappable (also to gem_object_pin) and
set it depending upon the context. Only bos that are brought into
the gtt due to an execbuffer call can be put into the unmappable
part of the gtt, everything else (especially pinned objects) need
to be put into the mappable part of the gtt.
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Add a mappable parameter to i915_gem_evict_something to distinguish
the two cases (non-restricted vs. mappable gtt allocations). No
functional changes because the mappable limit is set to the end of
the gtt currently.
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

In the event that we fail to dispatch the execbuffer, for example if
there is insufficient space on the ring, we were leaving the objects in
an inconsistent state. Notably they were marked as being in the GPU
write domain, but were not added to the ring or any list. This would
lead to inevitable oops:

[ 1010.522940] [drm:i915_gem_do_execbuffer] *ERROR* dispatch failed -16
[ 1010.523055] BUG: unable to handle kernel NULL pointer dereference at
0000000000000088
[ 1010.523097] IP: [<ffffffff8122d006>] i915_gem_flush_ring+0x26/0x140
[ 1010.523120] PGD 14cf2f067 PUD 14ce04067 PMD 0
[ 1010.523140] Oops: 0000 [#1] SMP
[ 1010.523154] last sysfs file: /sys/devices/virtual/vc/vcsa2/uevent
[ 1010.523173] CPU 0
[ 1010.523183] Pid: 716, comm: X Not tainted 2.6.36+ #34 LosLunas
CRB/SandyBridge Platform
[ 1010.523206] RIP: 0010:[<ffffffff8122d006>]  [<ffffffff8122d006>]
i915_gem_flush_ring+0x26/0x140
[ 1010.523233] RSP: 0018:ffff88014bf97cd8  EFLAGS: 00010296
[ 1010.523249] RAX: ffff88014e2d1808 RBX: 0000000000000000 RCX: 0000000000000000
[ 1010.523270] RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
[ 1010.523290] RBP: ffff88014e2d1000 R08: 0000000000000002 R09: 00000000400c645f
[ 1010.523311] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000002
[ 1010.523331] R13: ffff88014e29a000 R14: 00000000000000c8 R15: ffffffff8162eb28
[ 1010.523352] FS:  00007fc62379d700(0000) GS:ffff88001fc00000(0000) knlGS:0000000000000000
[ 1010.523375] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1010.523392] CR2: 0000000000000088 CR3: 000000014bf87000 CR4: 00000000000406f0
[ 1010.523412] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1010.523433] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1010.523454] Process X (pid: 716, threadinfo ffff88014bf96000, task ffff88014cc1ee40)
[ 1010.523475] Stack:
[ 1010.523483]  ffff88014d5199c0 0000000000000200 0000000000000000 ffff88014bcc6400
[ 1010.523509] <0> 0000000000000000 0000000000000001 ffff88014e29a000 ffff88014bcc6400
[ 1010.523537] <0> ffffffff8162eb28 ffffffff8122faa8 ffff88014e29a000 ffff88014bcc6400
[ 1010.523568] Call Trace:
[ 1010.523578]  [<ffffffff8122faa8>] ?  i915_gem_object_flush_gpu_write_domain+0x48/0x80
[ 1010.523601]  [<ffffffff8122fb8e>] ?  i915_gem_object_set_to_gtt_domain+0x2e/0xb0
[ 1010.523623]  [<ffffffff8123113b>] ?  i915_gem_set_domain_ioctl+0xdb/0x1f0
[ 1010.523644]  [<ffffffff8120a3f1>] ? drm_ioctl+0x3d1/0x460
[ 1010.523660]  [<ffffffff81231060>] ?  i915_gem_set_domain_ioctl+0x0/0x1f0
[ 1010.523682]  [<ffffffff81092618>] ?  vma_prio_tree_insert+0x28/0x120
[ 1010.523701]  [<ffffffff8109f379>] ? vma_link+0x99/0xf0
[ 1010.523717]  [<ffffffff810a111d>] ? mmap_region+0x1ed/0x4f0
[ 1010.523734]  [<ffffffff810c306f>] ? do_vfs_ioctl+0x9f/0x580
[ 1010.523750]  [<ffffffff810c3599>] ? sys_ioctl+0x49/0x80
[ 1010.523767]  [<ffffffff810022eb>] ?  system_call_fastpath+0x16/0x1b
[ 1010.523785] Code: 00 00 00 00 00 41 57 89 ce 41 56 41 55 41 54 45 89 c4 55 48 89 fd 53 48 89 d3 44 89 c2 48 89 df 4c 8d b3 c8 00 00 00 48 83 ec 18 <ff> 93 88 00 00 00 48 8b 83 c8 00 00 00 4c 8b bd 30 03 00 00 48
[ 1010.523946] RIP  [<ffffffff8122d006>] i915_gem_flush_ring+0x26/0x140
[ 1010.523966]  RSP <ffff88014bf97cd8>
[ 1010.523977] CR2: 0000000000000088
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Preparing the ringbuffer for adding new commands can fail (a timeout
whilst waiting for the GPU to catch up and free some space). So check
for any potential error before overwriting HEAD with new commands, and
propagate that error back to the user where possible.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

The ringbuffer keeps a pointer to the parent device, so we can use that
instead of passing around the pointer on the stack.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Keep the current interface but ignore the KM_type and use a stack based
approach.

The advantage is that we get rid of crappy code like:

	#define __KM_PTE			\
		(in_nmi() ? KM_NMI_PTE : 	\
		 in_irq() ? KM_IRQ_PTE :	\
		 KM_PTE0)

and in general can stop worrying about what context we're in and what kmap
slots might be appropriate for that.

The downside is that FRV kmap_atomic() gets more expensive.

For now we use a CPP trick suggested by Andrew:

  #define kmap_atomic(page, args...) __kmap_atomic(page)

to avoid having to touch all kmap_atomic() users in a single patch.

[ not compiled on:
  - mn10300: the arch doesn't actually build with highmem to begin with ]

[akpm@linux-foundation.org: coding-style fixes]
[akpm@linux-foundation.org: fix up drivers/gpu/drm/i915/intel_overlay.c]
Acked-by: NRik van Riel <riel@redhat.com>
Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
Acked-by: NChris Metcalf <cmetcalf@tilera.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Russell King <rmk@arm.linux.org.uk>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: David Miller <davem@davemloft.net>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Dave Airlie <airlied@linux.ie>
Cc: Li Zefan <lizf@cn.fujitsu.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

... to prevent flush processing of an idle (or even absent) ring.

This fixes a regression during suspend from 87acb0a5.
Reported-and-tested-by: NAlexey Fisher <bug-track@fisher-privat.net>
Tested-by: NPeter Clifton <pcjc2@cam.ac.uk>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

When the object has been written to by the gpu it remains on the ring
until its flush has been retired. However, when the object is moving to
the ring and the associated cache needs to be invalidated, we need to
perform the flush on the target ring, not the one it came from (which is
NULL in the reported case and so the flush was entirely absent).
Reported-by: NPeter Clifton <pcjc2@cam.ac.uk>
Reported-and-tested-by: NAlexey Fisher <bug-track@fisher-privat.net>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Whilst moving the code around in 9af90d19, I dropped the or'ing in of
new write domains which would zero out the write domain for a render
target if later reused as a source later in the batch. This meant that
we might drop a required flush before reading from the render target.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=31043
Reported-by: xunx.fang@intel.com
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Based on an original patch by Zhenyu Wang, this initializes the BLT ring for
SandyBridge and enables support for user execbuffers.

Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

If the userspace driver is using a constant relocation array with a
static buffer, they will pass the same relocation array back to the
kernel. So we *do* need to update the presumed offset value in those
relocations to reflect the current object so that they remain correct
with future batchbuffers and we avoid the necessity of having to suspend
execution and perform redundant relocations.

Fixes the regression introduced by 12f889c for applications using
absolute addressing on trees of buffer (i.e. the current consumers of
libdrm_intel.so).

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=30996Reported-by: NWang, Jinjin <jinjin.wang@intel.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

To handle retirements, we need per-ring tracking of active objects.
To handle evictions, we need global tracking of active objects.

As we enable more rings, rebuilding the global list from the individual
per-ring lists quickly grows tiresome and overly complicated. Tracking the
active objects in two lists is the lesser of two evils.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... by always initialising the empty ringbuffer it is always then safe
to check whether it is active.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

The most frequent relocation within a batchbuffer is a contiguous sequence
of vertex buffer relocations, for which we can virtually eliminate the
drm_gem_object_lookup() overhead by caching the last handle to object
translation.

In doing so we refactor the pin and relocate retry loop out of
do_execbuffer into its own helper function and so improve the error
paths.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

One of the primarily consumers of the i915 driver is X, a large signal
driven application. Frequently when writing into the buffers, there is a
pending signal which causes us not to take the interruptible lock but
then we need to take that same lock around the object unreference. By
rearranging the code to do the interruptible lock as the first check, we
can avoid the frequent additional locking around the unreference.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... to avoid the double acquisition along fast[er] paths.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... to avoid reacquiring it to drop the object reference count on
exit. Note we have to make sure we now drop (and reacquire) the lock
around acquiring the mm semaphore on the slow paths.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... in the hope that it makes the atomic fast paths more likely.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

After allocation a handle for the fresh object, we know that we can
safely drop the refcnt without triggering a free so we do not need the
mutex. Strangely, this mutex acquisition is the one that appears on
driver profiles.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Avoid an early eviction of the batch buffer into the uncached GTT
domain, and so do the relocation fixup in cacheable memory.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... perform an access validation check up front instead and copy them in
on-demand, during i915_gem_object_pin_and_relocate(). As around 20% of
the CPU overhead may be spent inside vmalloc for the relocation entries
when submitting an execbuffer [for x11perf -aa10text], the savings are
considerable and result in around a 10% throughput increase [for glyphs].
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Currently, if a batch buffer refers to an object with a pending flip,
then we sleep until that pending flip is completed (unpinned and
signalled). This is so that a flip can be queued and the user can
continue rendering to the backbuffer oblivious to whether the buffer is
still pinned as the scan out. (The kernel arbitrating at the last moment
to stall the batch and wait until the buffer is unpinned and replaced as
the front buffer.)

As we only have a queue depth of 1, we can simply wait for the current
pending flip to complete and continue rendering. We can achieve this
with a single WAIT_FOR_EVENT command inserted into the ring buffer prior
to executing the batch, *without* stalling the client.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... and do the same for pread.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org

Move the access control up from the fast paths, which are no longer
universally taken first, up into the caller. This then duplicates some
sanity checking along the slow paths, but is much simpler.
Tracked as CVE-2010-2962.
Reported-by: NKees Cook <kees@ubuntu.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org

Extend the error handling code with operations found in other nearby error
handling code

A simplified version of the sematic match that finds this problem is as
follows: (http://coccinelle.lip6.fr/)

// <smpl>
@r exists@
@r@
statement S1,S2,S3;
constant C1,C2,C3;
@@

*if (...)
 {... S1 return -C1;}
...
*if (...)
 {... when != S1
    return -C2;}
...
*if (...)
 {... S1 return -C3;}
// </smpl>
Signed-off-by: NJulia Lawall <julia@diku.dk>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org

The return from move_to_gtt_domain() may indicate a pending signal which
needs to handled as opposed to an actual error, for instance, so report
the original return value rather than forcing an EINVAL.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

When the GPU is reset, the fence registers are invalidated, so release
the objects and clear them out.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=30083Reported-by: NSitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Only drm/i915 does the bookkeeping that makes the information useful,
and the information maintained is driver specific, so move it out of the
core and into its single user.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Dave Airlie <airlied@redhat.com>

There were lots of places being inconsistent since handle count
looked like a kref but it really wasn't.

Fix this my just making handle count an atomic on the object,
and have it increase the normal object kref.

Now i915/radeon/nouveau drivers can drop the normal reference on
userspace object creation, and have the handle hold it.

This patch fixes a memory leak or corruption on unload, because
the driver had no way of knowing if a handle had been actually
added for this object, and the fbcon object needed to know this
to clean itself up properly.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDave Airlie <airlied@redhat.com>

At that point as the object is no longer in any GPU write domain it must
not be on the list, so the list_del() is redundant.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>