After reordering the sequence of relocating objects, commit 6fe4f140,
we can no longer rely on seeing all reloc targets prior to performing
the relocation. As a result we were ignoring the need to flush objects
from the render cache and invalidate the sampler caches, resulting in
rendering glitches. So we need to clear the relocation domains earlier.
Reported-by: NLinus Torvalds <torvalds@linux-foundation.org>
Tested-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

On the fault path, commit 6fe4f140 introduction a regression whereby it
changed the sequence of the objects but continued to use the original
ordering of relocation entries. The result was that incorrect GTT offsets
were being fed into the execbuffer causing lots of misrendering and
potential hangs.
Reported-by: NLinus Torvalds <torvalds@linux-foundation.org>
Tested-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

As the mappable portion of the aperture is always a small subset at the
start of the GTT, it is allocated preferentially by drm_mm. This is
useful in case we ever need to map an object later. However, if you have
a large object that can consume the entire mappable region of the
GTT this prevents the batchbuffer from fitting and so causing an error.
Instead allocate all those that require a mapping up front in order to
improve the likelihood of finding sufficient space to bind them.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Before releasing the lock in order to copy the relocation list from user
pages, we need to drop all the object references as another thread may
usurp and execute another batchbuffer before we reacquire the lock.
However, the code was buggy and failed to clear the list...
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org

... in order to avoid a BUG() and potential unbounded waits.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

We need to ensure that writes through the GTT land before any
modification to the MMIO registers and so must impose a mandatory write
barrier when flushing the GTT domain. This was revealed by relaxing the
write ordering by experimentally mapping the registers and the GATT as
write-combining.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

The relative-to-general state default is useless as it means having to
rewrite the streaming kernels for each batch. Relative-to-surface is
more useful, as that stream usually needs to be rewritten for each
batch. And absolute addressing mode, vital if you start streaming
state, is also only available by adjusting the register...
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

As we provide a list of all objects that will be accessed from the
batchbuffer, we can build a lut of the handles associated with those
objects for this invocation and use that to avoid the overhead of
looking up those objects again for every relocation.

The cost of building and searching a small hash table is much less than
that of acquiring a spinlock, searching a radix tree and manipulating an
atomic refcnt per relocation.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Userspace should not have been declaring that it needed fenced GPU
access with gen4+ as those GPUs have no fenced commands, but to be on
the safe side it is easier to ignore userspace in case they did.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

The bulk of the change is to convert the growing list of rings into an
array so that the relationship between the rings and the semaphore sync
registers can be easily computed.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

With this change, every batchbuffer can use all available fences (save
pinned and scanout, of course) without ever stalling the gpu!

In theory. Currently the actual pipelined update of the register is
disabled due to some stability issues. However, just the deferred update
is a significant win.

Based on a series of patches by Daniel Vetter.

The premise is that before every access to a buffer through the GTT we
have to declare whether we need a register or not. If the access is by
the GPU, a pipelined update to the register is made via the ringbuffer,
and we track the last seqno of the batches that access it. If by the
CPU we wait for the last GPU access and update the register (either
to clear or to set it for the current buffer).

One advantage of being able to pipeline changes is that we can defer the
actual updating of the fence register until we first need to access the
object through the GTT, i.e. we can eliminate the stall on set_tiling.
This is important as the userspace bo cache does not track the tiling
status of active buffers which generate frequent stalls on gen3 when
enabling tiling for an already bound buffer.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

This makes the various rings more consistent by removing the anomalous
handing of the rendering ring execbuffer dispatch.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

As the error occurred on the current object, it means that its state was
not changed and so it should be excluded from the unwind.
Reported-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Besides the minimal improvement in reducing the execbuffer overhead, the
real benefit is clarifying a few routines.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

A number of dragons have been seen lurking within the execbuffer code.
The first step is then to isolate them from the rest and begin to
scrutinise them in depth. Suggested by Daniel Vetter.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>