To handle retirements, we need per-ring tracking of active objects.
To handle evictions, we need global tracking of active objects.

As we enable more rings, rebuilding the global list from the individual
per-ring lists quickly grows tiresome and overly complicated. Tracking the
active objects in two lists is the lesser of two evils.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... by always initialising the empty ringbuffer it is always then safe
to check whether it is active.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

The most frequent relocation within a batchbuffer is a contiguous sequence
of vertex buffer relocations, for which we can virtually eliminate the
drm_gem_object_lookup() overhead by caching the last handle to object
translation.

In doing so we refactor the pin and relocate retry loop out of
do_execbuffer into its own helper function and so improve the error
paths.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

One of the primarily consumers of the i915 driver is X, a large signal
driven application. Frequently when writing into the buffers, there is a
pending signal which causes us not to take the interruptible lock but
then we need to take that same lock around the object unreference. By
rearranging the code to do the interruptible lock as the first check, we
can avoid the frequent additional locking around the unreference.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... to avoid the double acquisition along fast[er] paths.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... to avoid reacquiring it to drop the object reference count on
exit. Note we have to make sure we now drop (and reacquire) the lock
around acquiring the mm semaphore on the slow paths.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... in the hope that it makes the atomic fast paths more likely.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

After allocation a handle for the fresh object, we know that we can
safely drop the refcnt without triggering a free so we do not need the
mutex. Strangely, this mutex acquisition is the one that appears on
driver profiles.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Avoid an early eviction of the batch buffer into the uncached GTT
domain, and so do the relocation fixup in cacheable memory.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... perform an access validation check up front instead and copy them in
on-demand, during i915_gem_object_pin_and_relocate(). As around 20% of
the CPU overhead may be spent inside vmalloc for the relocation entries
when submitting an execbuffer [for x11perf -aa10text], the savings are
considerable and result in around a 10% throughput increase [for glyphs].
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Currently, if a batch buffer refers to an object with a pending flip,
then we sleep until that pending flip is completed (unpinned and
signalled). This is so that a flip can be queued and the user can
continue rendering to the backbuffer oblivious to whether the buffer is
still pinned as the scan out. (The kernel arbitrating at the last moment
to stall the batch and wait until the buffer is unpinned and replaced as
the front buffer.)

As we only have a queue depth of 1, we can simply wait for the current
pending flip to complete and continue rendering. We can achieve this
with a single WAIT_FOR_EVENT command inserted into the ring buffer prior
to executing the batch, *without* stalling the client.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... and do the same for pread.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org

Move the access control up from the fast paths, which are no longer
universally taken first, up into the caller. This then duplicates some
sanity checking along the slow paths, but is much simpler.
Tracked as CVE-2010-2962.
Reported-by: NKees Cook <kees@ubuntu.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org

Extend the error handling code with operations found in other nearby error
handling code

A simplified version of the sematic match that finds this problem is as
follows: (http://coccinelle.lip6.fr/)

// <smpl>
@r exists@
@r@
statement S1,S2,S3;
constant C1,C2,C3;
@@

*if (...)
 {... S1 return -C1;}
...
*if (...)
 {... when != S1
    return -C2;}
...
*if (...)
 {... S1 return -C3;}
// </smpl>
Signed-off-by: NJulia Lawall <julia@diku.dk>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org

The return from move_to_gtt_domain() may indicate a pending signal which
needs to handled as opposed to an actual error, for instance, so report
the original return value rather than forcing an EINVAL.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

When the GPU is reset, the fence registers are invalidated, so release
the objects and clear them out.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=30083Reported-by: NSitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Only drm/i915 does the bookkeeping that makes the information useful,
and the information maintained is driver specific, so move it out of the
core and into its single user.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Dave Airlie <airlied@redhat.com>

There were lots of places being inconsistent since handle count
looked like a kref but it really wasn't.

Fix this my just making handle count an atomic on the object,
and have it increase the normal object kref.

Now i915/radeon/nouveau drivers can drop the normal reference on
userspace object creation, and have the handle hold it.

This patch fixes a memory leak or corruption on unload, because
the driver had no way of knowing if a handle had been actually
added for this object, and the fbcon object needed to know this
to clean itself up properly.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDave Airlie <airlied@redhat.com>

At that point as the object is no longer in any GPU write domain it must
not be on the list, so the list_del() is redundant.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... and check more regularly.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Just reschedule the retire requests again if the device is currently
busy. The request list will be pruned along other paths so will never
grow unbounded and so we can afford to miss the occasional pruning.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Replaced by tracepoints.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

This has bitrotted through inuse and superseded by tracing and debugfs.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

With multiple rings generating requests independently, the outstanding
requests must also be track independently.
Reported-by: NWang Jinjin <jinjin.wang@intel.com>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=30380Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Introduced by 48b956c5, I had thought I had already fixed this. Oh well.
Reported-by: NSitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Daniel Vetter pointed out that in this case is would be clearer and
cleaner to use a spinlock instead of a mutex to protect the per-file
request list manipulation. Make it so.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... and combine it with the wedged completion handler.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Owain Ainsworth reported an issue between the interaction of the
hangcheck and userspace immediately (and permanently) falling back to
s/w rasterisation. In order to break the mutex and begin resetting the
GPU, we must abort the current operation (usually within the wait) and
climb sufficiently far back up the call chain to drop the mutex. In his
implementation, Owain has a loop within the ioctl handler to detect the
hang and then sleep until the error handler has run. I've chosen to
return to userspace and report an EAGAIN which should trigger the
userspace ioctl handler to repeat the call (simply because it felt less
invasive...). Before hitting a wedged GPU, we then wait upon completion
of the error handler.
Reported-by: NOwain G. Ainsworth <zerooa@googlemail.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Avoid cause latencies in other clients by not taking the global struct
mutex and moving the per-client request manipulation a local per-client
mutex. For example, this allows a compositor to schedule a page-flip
(through X) whilst an OpenGL application is monopolising the GPU.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

We need to drain the pending flips prior to disabling the pipe during
modeset, and these need to be done in an uninterruptible fashion.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

This is already performed with the pipelined flush, so by the time we
schedule the flush in the page-flip, the ring is NULL and we OOPs
instead.
Reported-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

A minor typo caused a single fence register to be incorrectly
programmed, resulting in occassional tiling corruption.
Reported-and-tested-by: NHans de Bruin <bruinjm@xs4all.nl>
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=18962Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org

We are not currently using it as intended, so remove the complication.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Otherwise we will hit a list handling assertion when moving the object
to the inactive list.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

During i915_gem_create_mmap_offset() if the subsystem reports an error
code, use it.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Keep a list of pinned objects and display it via debugfs. Now all
objects that exist in the GTT are always tracked on one of the
active, flushing, inactive or pinned lists.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

If we have queued a page flip on the current fb and then request a mode
change, wait until the page flip completes before performing the new
request.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>