Prepared substructure rps for RPS related state. autoenable_work is
used for RC6 too hence it is defined outside rps structure. As we do
this lot many functions are refactored to use intel_rps *rps to access
rps related members. Hence renamed intel_rps_client pointer variables
to rps_client in various functions.

v2: Rebase.

v3: s/pm/gt_pm (Chris)
Refactored access to rps structure by declaring struct intel_rps * in
many functions.
Signed-off-by: NSagar Arun Kamble <sagar.a.kamble@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Imre Deak <imre.deak@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Radoslaw Szwichtenberg <radoslaw.szwichtenberg@intel.com> #1
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/1507360055-19948-9-git-send-email-sagar.a.kamble@intel.comAcked-by: NImre Deak <imre.deak@intel.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20171010213010.7415-8-chris@chris-wilson.co.uk

Add another perma-pinned context for using for preemption at any time.
We cannot just reuse the existing kernel context, as first and foremost
we need to ensure that we can preempt the kernel context itself, so
require a distinct context id. Similar to the kernel context, we may
want to interrupt execution and switch to the preempt context at any
time, and so it needs to be permanently pinned and available.

To compensate for yet another permanent allocation, we shrink the
existing context and the new context by reducing their ringbuffer to the
minimum.

v2: Assert that we never allocate a request from the preemption context.
v3: Limit perma-pin to engines that may preempt.
v4: Onion cleanup for early driver death
v5: Onion ordering in main driver cleanup as well.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NMichał Winiarski <michal.winiarski@intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-4-chris@chris-wilson.co.uk

We use INT_MIN to denote the priority of a request that has not been
submitted to the scheduler; we treat INT_MIN as an invalid priority and
initialise the request to it. Give the value a name so it stands out.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20170928193910.17988-3-chris@chris-wilson.co.ukReviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com

No users now outside of i915_wait_request(), so we can make it private to
i915_gem_request.c, and assume the caller knows the seqno. In the
process, also remove i915_gem_request_started() as that was only ever
used by i915_spin_request().
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20170922120333.25535-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

After we see our target seqno has been completed by the hw, we need to
confirm that it still matches the request (as it may have been preempted
before the spin completes). If the request no longer matches the target
seqno, we need to restart the wait to reacquire that seqno.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20170921210903.18337-1-chris@chris-wilson.co.ukReviewed-by: NMichał Winiarski <michal.winiarski@intel.com>

In a synchronous setup, we may retire the last request before we
complete allocating the next request. As the last request is retired, we
queue a timer to mark the device as idle, and promptly have to execute
ad cancel that timer once we complete allocating the request and need to
keep the device awake. If we rearrange the mark_busy() to occur before
we retire the previous request, we can skip this ping-pong.

v2: Joonas pointed out that unreserve_seqno() was now doing more than
doing seqno handling and should be renamed to reflect its wider purpose.
That also highlighted the new asymmetry with reserve_seqno(), so fixup
that and rename both to [un]reserve_engine().
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20170817144719.10968-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

During our selftests, we try reseting the GPU tens of thousands of
times, flooding the dmesg with our reset spam drowning out any potential
warnings. Add an option to i915_reset()/i915_reset_engine() to specify a
quiet reset for selftesting.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20170721123238.16428-19-chris@chris-wilson.co.ukReviewed-by: NMichel Thierry <michel.thierry@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Since we make call i915_gem_context_mark_guilty() concurrently when
resetting different engines in parallel, we need to make sure that our
updates are safe for the unlocked access.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Michel Thierry <michel.thierry@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20170721123238.16428-12-chris@chris-wilson.co.ukSigned-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

intel_engine_init_globa_seqno() may be called from an uncontrolled
set-wedged path where we have given up waiting for broken hw and declare
it defunct. Along that path, any sanity checks that the hw is idle
before we adjust its state will expectedly fail, so we simply cannot.
Instead of asserting inside init_global_seqno, we move them to the
normal caller reset_all_global_seqno() as it handles runtime seqno
wraparound.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20170721123238.16428-8-chris@chris-wilson.co.ukSigned-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Once a client has requested a waitboost, we keep that waitboost active
until all clients are no longer waiting. This is because we don't
distinguish which waiter deserves the boost. However, with the advent of
fence signaling, the signaler threads appear as waiters to the RPS
interrupt handler. So instead of using a single boolean to track when to
keep the waitboost active, use a counter of all outstanding waitboosted
requests.

At this point, I have removed all vestiges of the rate limiting on
clients. Whilst this means that compositors should remain more fluid,
it also means that boosts are more prevalent. See commit b29c19b6
("drm/i915: Boost RPS frequency for CPU stalls") for a longer discussion
on the pros and cons of both approaches.

A drawback of this implementation is that it requires constant request
submission to keep the waitboost trimmed (as it is now cancelled when the
request is completed). This will be fine for a busy system, but near
idle the boosts may be kept for longer than desired (effectively tens of
vblanks worstcase) and there is a reliance on rc6 instead.

v2: Remove defunct rps.client_lock
Reported-by: NMichał Winiarski <michal.winiarski@intel.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: NMichał Winiarski <michal.winiarski@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170628123548.9236-1-chris@chris-wilson.co.uk

We need to keep track of the last location we ask the hw to read up to
(RING_TAIL) separately from our last write location into the ring, so
that in the event of a GPU reset we do not tell the HW to proceed into
a partially written request (which can happen if that request is waiting
for an external signal before being executed).

v2: Refactor intel_ring_reset() (Mika)

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=100144
Testcase: igt/gem_exec_fence/await-hang
Fixes: 821ed7df ("drm/i915: Update reset path to fix incomplete requests")
Fixes: d55ac5bf ("drm/i915: Defer transfer onto execution timeline to actual hw submission")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170425130049.26147-1-chris@chris-wilson.co.ukReviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
(cherry picked from commit e6ba9992)
Signed-off-by: NJani Nikula <jani.nikula@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170615131129.3061-1-chris@chris-wilson.co.uk

Originally we would enable and disable the breadcrumb interrupt
immediately on demand. This was slow enough to have a large impact
(>30%) on tasks that hopped between engines. However, by using a shadow
to keep the irq alive for an extra interrupt (see commit 67b807a8
("drm/i915: Delay disabling the user interrupt for breadcrumbs")) and
by recently reducing the cost in adding ourselves to the signal tree, we
no longer need to spin-request during await_request to avoid delays in
throughput tests. Without the earlier patches to stop the wakeup when
signaling if the irq was already active, we saw no improvement in
execbuf overhead (and corresponding contention in other clients) despite
the removal of the spinner in a simple test like glxgears. This means
there will be scenarios where now we spend longer enabling the interrupt
than we would have spent spinning, but these are not likely to have as
noticeable an impact as the high frequency test cases (where there
should not be any regression).

Ulterior motive: generalising the engine->sync_to to handle different
types of semaphores and non-semaphores.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170608111405.16466-4-chris@chris-wilson.co.uk

Setting up the irq to signal the request completion takes a finite
amount of time, during which it is possible that the request already
completed. Check afterwards, just in case, so that we can respond
immediately.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170608111405.16466-1-chris@chris-wilson.co.uk

Passing NULL ctx to request_alloc would lead to null-ptr-deref.

v2: Let's not replace the comment with a BUG_ON
Signed-off-by: NMichał Winiarski <michal.winiarski@intel.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: http://patchwork.freedesktop.org/patch/msgid/20170523102400.9614-1-michal.winiarski@intel.comSigned-off-by: NChris Wilson <chris@chris-wilson.co.uk>

All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.

Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.

There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).

v2: Avoid use-after-free when deleting a depleted priolist

v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: NMichał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NMichał Winiarski <michal.winiarski@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk

Since unifying ringbuffer/execlist submission to use
engine->pin_context, we ensure that the intel_ring is available before
we start constructing the request. We can therefore move the assignment
of the request->ring to the central i915_gem_request_alloc() and not
require it in every engine->request_alloc() callback. Another small step
towards simplification (of the core, but at a cost of handling error
pointers in less important callers of engine->pin_context).

v2: Rearrange a few branches to reduce impact of PTR_ERR() on gcc's code
generation.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Oscar Mateo <oscar.mateo@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NOscar Mateo <oscar.mateo@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170504093308.4137-1-chris@chris-wilson.co.uk

Rather than use a global modparam, we can just check to see if the
engine has semaphores configured upon it.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170503093924.5320-7-chris@chris-wilson.co.uk

As we may unwind the requests, even though the request we are awaiting
has a global_seqno that seqno may be revoked during the await and so we
can not reliably use it as a barrier for all future awaits on the same
timeline.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: NMichał Winiarski <michal.winiarski@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170503093924.5320-6-chris@chris-wilson.co.uk

With the addition of the inter-context intel_time.sync map, having a
very similar sync_seqno[] is confusing. Aide the reader by denoting that
this is a pre-allocated array for storing semaphore sync points wrt to
the global seqno.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170503093924.5320-5-chris@chris-wilson.co.uk

Track the latest fence waited upon on each context, and only add a new
asynchronous wait if the new fence is more recent than the recorded
fence for that context. This requires us to filter out unordered
timelines, which are noted by DMA_FENCE_NO_CONTEXT. However, in the
absence of a universal identifier, we have to use our own
i915->mm.unordered_timeline token.

v2: Throw around the debug crutches
v3: Inline the likely case of the pre-allocation cache being full.
v4: Drop the pre-allocation support, we can lose the most recent fence
in case of allocation failure -- it just means we may emit more awaits
than strictly necessary but will not break.
v5: Trim allocation size for leaf nodes, they only need an array of u32
not pointers.
v6: Create mock_timeline to tidy selftest writing
v7: s/intel_timeline_sync_get/intel_timeline_sync_is_later/ (Tvrtko)
v8: Prune the stale sync points when we idle.
v9: Include a small benchmark in the kselftests
v10: Separate the idr implementation into its own compartment. (Tvrkto)
v11: Refactor igt_sync kselftests to avoid deep nesting (Tvrkto)
v12: __sync_leaf_idx() to assert that p->height is 0 when checking leaves
v13: kselftests to investigate struct i915_syncmap itself (Tvrtko)
v14: Foray into ascii art graphs
v15: Take into account that the random lookup/insert does 2 prng calls,
not 1, when benchmarking, and use for_each_set_bit() (Tvrtko)
v16: Improved ascii art
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170503093924.5320-4-chris@chris-wilson.co.uk

Currently we filter out repeated use of the same timeline in the low
level i915_gem_request_await_request(), after having added the
dependency on the old request. However, we can lift this to
i915_gem_request_await_dma_fence() (before the dependency is added)
using the observation that requests along the same timeline are
explicitly ordered via i915_add_request (along with the dependencies).
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170503093924.5320-3-chris@chris-wilson.co.uk

By first unwrapping an incoming fence-array into its child fences, we
can simplify the internal branching, and so avoid triggering a potential
bug in the next patch when not squashing the child fences on the same
timeline.

It will also have the advantage of keeping the (top-level) fence arrays
out of any fence/timeline caching since these are unordered timelines
but with a random context id.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170503093924.5320-2-chris@chris-wilson.co.uk

Although we do check the completion-status of the request before
actually adding a wait on it (either to its submit fence or its
completion dma-fence), we currently do not check before adding it to the
dependency lists.

In fact, without checking for a completed request we may try to use the
signaler after it has been retired and its dependency tree freed:

[   60.044057] BUG: KASAN: use-after-free in __list_add_valid+0x1d/0xd0 at addr ffff880348c9e6a0
[   60.044118] Read of size 8 by task gem_exec_fence/530
[   60.044164] CPU: 1 PID: 530 Comm: gem_exec_fence Tainted: G            E   4.11.0-rc7+ #46
[   60.044226] Hardware name: ��������������������������������� ���������������������������������/���������������������������������, BIOS RYBDWi35.86A.0246.2
[   60.044290] Call Trace:
[   60.044337]  dump_stack+0x4d/0x6a
[   60.044383]  kasan_object_err+0x21/0x70
[   60.044435]  kasan_report+0x225/0x4e0
[   60.044488]  ? __list_add_valid+0x1d/0xd0
[   60.044534]  ? kasan_kmalloc+0xad/0xe0
[   60.044587]  __asan_load8+0x5e/0x70
[   60.044639]  __list_add_valid+0x1d/0xd0
[   60.044788]  __i915_priotree_add_dependency+0x67/0x130 [i915]
[   60.044895]  i915_gem_request_await_request+0xa8/0x370 [i915]
[   60.044974]  i915_gem_request_await_dma_fence+0x129/0x140 [i915]
[   60.045049]  i915_gem_do_execbuffer.isra.37+0xb0a/0x26b0 [i915]
[   60.045077]  ? save_stack+0xb1/0xd0
[   60.045105]  ? save_stack_trace+0x1b/0x20
[   60.045132]  ? save_stack+0x46/0xd0
[   60.045158]  ? kasan_kmalloc+0xad/0xe0
[   60.045184]  ? __kmalloc+0xd8/0x670
[   60.045229]  ? drm_ioctl+0x359/0x640 [drm]
[   60.045256]  ? SyS_ioctl+0x41/0x70
[   60.045330]  ? i915_vma_move_to_active+0x540/0x540 [i915]
[   60.045360]  ? tty_insert_flip_string_flags+0xa1/0xf0
[   60.045387]  ? tty_flip_buffer_push+0x63/0x70
[   60.045414]  ? remove_wait_queue+0xa9/0xc0
[   60.045441]  ? kasan_unpoison_shadow+0x35/0x50
[   60.045467]  ? kasan_kmalloc+0xad/0xe0
[   60.045494]  ? kasan_check_write+0x14/0x20
[   60.045568]  i915_gem_execbuffer2+0xdb/0x2a0 [i915]
[   60.045616]  drm_ioctl+0x359/0x640 [drm]
[   60.045705]  ? i915_gem_execbuffer+0x5a0/0x5a0 [i915]
[   60.045751]  ? drm_version+0x150/0x150 [drm]
[   60.045778]  ? compat_start_thread+0x60/0x60
[   60.045805]  ? plist_del+0xda/0x1a0
[   60.045833]  do_vfs_ioctl+0x12e/0x910
[   60.045860]  ? ioctl_preallocate+0x130/0x130
[   60.045886]  ? pci_mmcfg_check_reserved+0xc0/0xc0
[   60.045913]  ? vfs_write+0x196/0x240
[   60.045939]  ? __fget_light+0xa7/0xc0
[   60.045965]  SyS_ioctl+0x41/0x70
[   60.045991]  entry_SYSCALL_64_fastpath+0x17/0x98
[   60.046017] RIP: 0033:0x7feb2baefc47
[   60.046042] RSP: 002b:00007fff56d28e58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[   60.046075] RAX: ffffffffffffffda RBX: 00007fff56d290a8 RCX: 00007feb2baefc47
[   60.046102] RDX: 00007fff56d29050 RSI: 00000000c0406469 RDI: 0000000000000003
[   60.046129] RBP: 00007fff56d29050 R08: 000055ecc4cd27d0 R09: 00007feb2bda8600
[   60.046154] R10: 0000000000000073 R11: 0000000000000246 R12: 00000000c0406469
[   60.046177] R13: 0000000000000003 R14: 000000000000000f R15: 0000000000000099
[   60.046203] Object at ffff880348c9e680, in cache i915_dependency size: 64
[   60.046225] Allocated:
[   60.046246] PID = 530
[   60.046269]  save_stack_trace+0x1b/0x20
[   60.046292]  save_stack+0x46/0xd0
[   60.046318]  kasan_kmalloc+0xad/0xe0
[   60.046343]  kasan_slab_alloc+0x12/0x20
[   60.046368]  kmem_cache_alloc+0xab/0x650
[   60.046445]  i915_gem_request_await_request+0x88/0x370 [i915]
[   60.046559]  i915_gem_request_await_dma_fence+0x129/0x140 [i915]
[   60.046705]  i915_gem_do_execbuffer.isra.37+0xb0a/0x26b0 [i915]
[   60.046849]  i915_gem_execbuffer2+0xdb/0x2a0 [i915]
[   60.046936]  drm_ioctl+0x359/0x640 [drm]
[   60.046987]  do_vfs_ioctl+0x12e/0x910
[   60.047038]  SyS_ioctl+0x41/0x70
[   60.047090]  entry_SYSCALL_64_fastpath+0x17/0x98
[   60.047139] Freed:
[   60.047179] PID = 530
[   60.047223]  save_stack_trace+0x1b/0x20
[   60.047269]  save_stack+0x46/0xd0
[   60.047317]  kasan_slab_free+0x72/0xc0
[   60.047366]  kmem_cache_free+0x39/0x160
[   60.047512]  i915_gem_request_retire+0x83f/0x930 [i915]
[   60.047657]  i915_gem_request_alloc+0x166/0x600 [i915]
[   60.047799]  i915_gem_do_execbuffer.isra.37+0xad8/0x26b0 [i915]
[   60.047897]  i915_gem_execbuffer2+0xdb/0x2a0 [i915]
[   60.047942]  drm_ioctl+0x359/0x640 [drm]
[   60.047968]  do_vfs_ioctl+0x12e/0x910
[   60.047993]  SyS_ioctl+0x41/0x70
[   60.048019]  entry_SYSCALL_64_fastpath+0x17/0x98
[   60.048044] Memory state around the buggy address:
[   60.048066]  ffff880348c9e580: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
[   60.048105]  ffff880348c9e600: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
[   60.048138] >ffff880348c9e680: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
[   60.048170]                                ^
[   60.048191]  ffff880348c9e700: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
[   60.048225]  ffff880348c9e780: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc

Note to hit the use-after-free requires us to be passed back a request
via a fence-array, that is from explicit fencing accumulated into a
sync-file fence-array.

Fixes: 52e54209 ("drm/i915/scheduler: Record all dependencies upon request construction")
Testcase: igt/gem_exec_fence/expired-history
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NMichał Winiarski <michal.winiarski@intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170422081537.6468-1-chris@chris-wilson.co.uk
(cherry picked from commit ade0b0c9)
Signed-off-by: NJani Nikula <jani.nikula@intel.com>

If we are enabling the breadcrumbs signaling prior to submitting the
request, we know that we cannot have missed the interrupt and can
therefore skip immediately waking the signaler to check.

This reduces a significant chunk of the __i915_gem_request_submit()
overhead for inter-engine synchronisation, for example in gem_exec_whisper.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170426080659.28771-1-chris@chris-wilson.co.ukReviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>

We need to keep track of the last location we ask the hw to read up to
(RING_TAIL) separately from our last write location into the ring, so
that in the event of a GPU reset we do not tell the HW to proceed into
a partially written request (which can happen if that request is waiting
for an external signal before being executed).

v2: Refactor intel_ring_reset() (Mika)

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=100144
Testcase: igt/gem_exec_fence/await-hang
Fixes: 821ed7df ("drm/i915: Update reset path to fix incomplete requests")
Fixes: d55ac5bf ("drm/i915: Defer transfer onto execution timeline to actual hw submission")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170425130049.26147-1-chris@chris-wilson.co.ukReviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>

Although we do check the completion-status of the request before
actually adding a wait on it (either to its submit fence or its
completion dma-fence), we currently do not check before adding it to the
dependency lists.

In fact, without checking for a completed request we may try to use the
signaler after it has been retired and its dependency tree freed:

[   60.044057] BUG: KASAN: use-after-free in __list_add_valid+0x1d/0xd0 at addr ffff880348c9e6a0
[   60.044118] Read of size 8 by task gem_exec_fence/530
[   60.044164] CPU: 1 PID: 530 Comm: gem_exec_fence Tainted: G            E   4.11.0-rc7+ #46
[   60.044226] Hardware name: ��������������������������������� ���������������������������������/���������������������������������, BIOS RYBDWi35.86A.0246.2
[   60.044290] Call Trace:
[   60.044337]  dump_stack+0x4d/0x6a
[   60.044383]  kasan_object_err+0x21/0x70
[   60.044435]  kasan_report+0x225/0x4e0
[   60.044488]  ? __list_add_valid+0x1d/0xd0
[   60.044534]  ? kasan_kmalloc+0xad/0xe0
[   60.044587]  __asan_load8+0x5e/0x70
[   60.044639]  __list_add_valid+0x1d/0xd0
[   60.044788]  __i915_priotree_add_dependency+0x67/0x130 [i915]
[   60.044895]  i915_gem_request_await_request+0xa8/0x370 [i915]
[   60.044974]  i915_gem_request_await_dma_fence+0x129/0x140 [i915]
[   60.045049]  i915_gem_do_execbuffer.isra.37+0xb0a/0x26b0 [i915]
[   60.045077]  ? save_stack+0xb1/0xd0
[   60.045105]  ? save_stack_trace+0x1b/0x20
[   60.045132]  ? save_stack+0x46/0xd0
[   60.045158]  ? kasan_kmalloc+0xad/0xe0
[   60.045184]  ? __kmalloc+0xd8/0x670
[   60.045229]  ? drm_ioctl+0x359/0x640 [drm]
[   60.045256]  ? SyS_ioctl+0x41/0x70
[   60.045330]  ? i915_vma_move_to_active+0x540/0x540 [i915]
[   60.045360]  ? tty_insert_flip_string_flags+0xa1/0xf0
[   60.045387]  ? tty_flip_buffer_push+0x63/0x70
[   60.045414]  ? remove_wait_queue+0xa9/0xc0
[   60.045441]  ? kasan_unpoison_shadow+0x35/0x50
[   60.045467]  ? kasan_kmalloc+0xad/0xe0
[   60.045494]  ? kasan_check_write+0x14/0x20
[   60.045568]  i915_gem_execbuffer2+0xdb/0x2a0 [i915]
[   60.045616]  drm_ioctl+0x359/0x640 [drm]
[   60.045705]  ? i915_gem_execbuffer+0x5a0/0x5a0 [i915]
[   60.045751]  ? drm_version+0x150/0x150 [drm]
[   60.045778]  ? compat_start_thread+0x60/0x60
[   60.045805]  ? plist_del+0xda/0x1a0
[   60.045833]  do_vfs_ioctl+0x12e/0x910
[   60.045860]  ? ioctl_preallocate+0x130/0x130
[   60.045886]  ? pci_mmcfg_check_reserved+0xc0/0xc0
[   60.045913]  ? vfs_write+0x196/0x240
[   60.045939]  ? __fget_light+0xa7/0xc0
[   60.045965]  SyS_ioctl+0x41/0x70
[   60.045991]  entry_SYSCALL_64_fastpath+0x17/0x98
[   60.046017] RIP: 0033:0x7feb2baefc47
[   60.046042] RSP: 002b:00007fff56d28e58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[   60.046075] RAX: ffffffffffffffda RBX: 00007fff56d290a8 RCX: 00007feb2baefc47
[   60.046102] RDX: 00007fff56d29050 RSI: 00000000c0406469 RDI: 0000000000000003
[   60.046129] RBP: 00007fff56d29050 R08: 000055ecc4cd27d0 R09: 00007feb2bda8600
[   60.046154] R10: 0000000000000073 R11: 0000000000000246 R12: 00000000c0406469
[   60.046177] R13: 0000000000000003 R14: 000000000000000f R15: 0000000000000099
[   60.046203] Object at ffff880348c9e680, in cache i915_dependency size: 64
[   60.046225] Allocated:
[   60.046246] PID = 530
[   60.046269]  save_stack_trace+0x1b/0x20
[   60.046292]  save_stack+0x46/0xd0
[   60.046318]  kasan_kmalloc+0xad/0xe0
[   60.046343]  kasan_slab_alloc+0x12/0x20
[   60.046368]  kmem_cache_alloc+0xab/0x650
[   60.046445]  i915_gem_request_await_request+0x88/0x370 [i915]
[   60.046559]  i915_gem_request_await_dma_fence+0x129/0x140 [i915]
[   60.046705]  i915_gem_do_execbuffer.isra.37+0xb0a/0x26b0 [i915]
[   60.046849]  i915_gem_execbuffer2+0xdb/0x2a0 [i915]
[   60.046936]  drm_ioctl+0x359/0x640 [drm]
[   60.046987]  do_vfs_ioctl+0x12e/0x910
[   60.047038]  SyS_ioctl+0x41/0x70
[   60.047090]  entry_SYSCALL_64_fastpath+0x17/0x98
[   60.047139] Freed:
[   60.047179] PID = 530
[   60.047223]  save_stack_trace+0x1b/0x20
[   60.047269]  save_stack+0x46/0xd0
[   60.047317]  kasan_slab_free+0x72/0xc0
[   60.047366]  kmem_cache_free+0x39/0x160
[   60.047512]  i915_gem_request_retire+0x83f/0x930 [i915]
[   60.047657]  i915_gem_request_alloc+0x166/0x600 [i915]
[   60.047799]  i915_gem_do_execbuffer.isra.37+0xad8/0x26b0 [i915]
[   60.047897]  i915_gem_execbuffer2+0xdb/0x2a0 [i915]
[   60.047942]  drm_ioctl+0x359/0x640 [drm]
[   60.047968]  do_vfs_ioctl+0x12e/0x910
[   60.047993]  SyS_ioctl+0x41/0x70
[   60.048019]  entry_SYSCALL_64_fastpath+0x17/0x98
[   60.048044] Memory state around the buggy address:
[   60.048066]  ffff880348c9e580: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
[   60.048105]  ffff880348c9e600: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
[   60.048138] >ffff880348c9e680: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
[   60.048170]                                ^
[   60.048191]  ffff880348c9e700: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
[   60.048225]  ffff880348c9e780: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc

Note to hit the use-after-free requires us to be passed back a request
via a fence-array, that is from explicit fencing accumulated into a
sync-file fence-array.

Fixes: 52e54209 ("drm/i915/scheduler: Record all dependencies upon request construction")
Testcase: igt/gem_exec_fence/expired-history
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NMichał Winiarski <michal.winiarski@intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170422081537.6468-1-chris@chris-wilson.co.uk

Introduce a new execobject.flag (EXEC_OBJECT_CAPTURE) that userspace may
use to indicate that it wants the contents of this buffer preserved in
the error state (/sys/class/drm/cardN/error) following a GPU hang
involving this batch.

Use this at your discretion, the contents of the error state. although
compressed, are allocated with GFP_ATOMIC (i.e. limited) and kept for all
eternity (until the error state is destroyed).

Based on an earlier patch by Ben Widawsky <ben@bwidawsk.net>
Testcase: igt/gem_exec_capture
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Ben Widawsky <ben@bwidawsk.net>
Cc: Matt Turner <mattst88@gmail.com>
Acked-by: NBen Widawsky <ben@bwidawsk.net>
Acked-by: NMatt Turner <mattst88@gmail.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170415093902.22581-1-chris@chris-wilson.co.uk

When we retire the last request on the ring, before we ever access that
ring again we know it will be completely idle and so we can advance the
ring->head fully to the end (i.e. ring->tail) and not just to the start
of the breadcrumb. This allows us to skip re-emitting the breadcrumb
after resetting the GPU if the ring was entirely idle. This prevents us
from overwriting a seqno wraparound by re-executing a stale breadcrumb,
i.e.
	submit_request(1)
	intel_engine_init_global_seqno(0)
	i915_reset()
would then leave 1 in the HWS, but the next request to execute would
also be with seqno 1. The sanity checks upon submission detect this as a
timewarp and explode. By setting the ring as empty, upon reset the HWS
is left as 0, leaving it consistent with the timeline.

v2: Fix check for deleting last element of list. We know that this
request is always the first element of the ring, so only if next
points back to the start will this be the only request in flight.
v3: Remove opencoding of list_is_last()
v4: Move the block to its own function for some clarity.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=100144
Testcase: igt/gem_exec_whisper/hang-*
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170406170028.26871-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

When we update the global seqno (on the engine timeline), we modify HW
state (both registers and mapped pages). As we do this, we should be
sure that the HW is idle and we are not causing a conflict. The caller
is supposed to wait_for_idle before calling us to update the seqno, so
let's assert they have and the engine is indeed idle.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: http://patchwork.freedesktop.org/patch/msgid/20170405153055.28123-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

We can merge the pair of loops over the engines and their timelines into
a single loop, making it easier to read and more consistent with the
commentary.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: http://patchwork.freedesktop.org/patch/msgid/20170330145041.9005-6-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

Having added the wait upon each engine to idle into the central
i915_gem_wait_for_idle(), we can remove the now redundant wait from
reset_all_global_seqno(). This has the advantage of removing the late
detection of an error (an engine still busy) which left the seqno reset
only partially complete (though it should be safe enough!).
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: http://patchwork.freedesktop.org/patch/msgid/20170330145041.9005-5-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

As we now distinguish everywhere that can call
i915_gem_retire_requests() following a successful wait_for_idle, we can
remove the duplication by moving that call into i915_gem_wait_for_idle()
itself.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: http://patchwork.freedesktop.org/patch/msgid/20170330145041.9005-3-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

Michał Winiarski pointed out that the debugging infrastructure (such as
trace_dma_fence_release) likes to pretty print the timeline name, long
after we have freed the timeline. Our timelines currently live as part of
the GTT (due to the strict ordering we currently use through each) which
belong to the context. We aim to free the context and release its
hardware resources as soon as we able to (i.e. when the last
fence/request using it has been signaled and retired). As the
.get_timeline_name is purely a debug feature, rather than extending the
lifetime of the context, or splitting it into many different release
phases just to keep the name around, replace the timeline name with a
constant after the fence has been signaled. This avoids the potential
use-after-free.
Reported-by: NKrzysztof Olinski <krzysztof.e.olinski@intel.com>
Fixes: 80b204bc ("drm/i915: Enable multiple timelines")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: <stable@vger.kernel.org> # v4.10+
Link: http://patchwork.freedesktop.org/patch/msgid/20170330111614.29757-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NMichał Winiarski <michal.winiarski@intel.com>
(cherry picked from commit 05506b5b)
Signed-off-by: NJani Nikula <jani.nikula@intel.com>

Michał Winiarski pointed out that the debugging infrastructure (such as
trace_dma_fence_release) likes to pretty print the timeline name, long
after we have freed the timeline. Our timelines currently live as part of
the GTT (due to the strict ordering we currently use through each) which
belong to the context. We aim to free the context and release its
hardware resources as soon as we able to (i.e. when the last
fence/request using it has been signaled and retired). As the
.get_timeline_name is purely a debug feature, rather than extending the
lifetime of the context, or splitting it into many different release
phases just to keep the name around, replace the timeline name with a
constant after the fence has been signaled. This avoids the potential
use-after-free.
Reported-by: NKrzysztof Olinski <krzysztof.e.olinski@intel.com>
Fixes: 80b204bc ("drm/i915: Enable multiple timelines")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: <stable@vger.kernel.org> # v4.10+
Link: http://patchwork.freedesktop.org/patch/msgid/20170330111614.29757-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NMichał Winiarski <michal.winiarski@intel.com>

Storing the position of the breadcrumb of the last retired request as
a separate last_retired_head is superfluous as we always copy that into
head prior to recalculation of the intel_ring.space.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170321102552.24357-1-chris@chris-wilson.co.uk

I915_RESET_IN_PROGRESS is being used for both signaling the requirement
to i915_mutex_lock_interruptible() to avoid taking the struct_mutex and
to instruct a waiter (already holding the struct_mutex) to perform the
reset. To allow for a little more coordination, split these two meaning
into a couple of distinct flags. I915_RESET_BACKOFF tells
i915_mutex_lock_interruptible() not to acquire the mutex and
I915_RESET_HANDOFF tells the waiter to call i915_reset().
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Acked-by: NMichel Thierry <michel.thierry@intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170316171305.12972-1-chris@chris-wilson.co.uk

During reset_all_global_seqno() on seqno rollover, we have to update the
HWS. This causes all in flight requests to be completed, so first we
wait. However, we were only waiting for the requests themselves to be
completed and clearing out the waiter rbtrees - what I had missed was
the extra reference in execlists->port[]. Since commit fe9ae7a3
("drm/i915/execlists: Detect an out-of-order context switch") we can
detect when the request is retired before the context switch interrupt
is completed. The impact should be neglible outside of debugging.

Testcase: igt/gem_exec_whisper
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170303121947.20482-1-chris@chris-wilson.co.ukReviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>

Adding to the tail of the client request list as the only other user is
in the throttle ioctl that iterates forwards over the list. It only
needs protection against deletion of a request as it reads it, it simply
won't see a new request added to the end of the list, or it would be too
early and rejected. We can further reduce the number of spinlocks
required when throttling by removing stale requests from the client_list
as we throttle.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170302122525.19675-1-chris@chris-wilson.co.uk

assert_spin_locked() becomes an unconditionally compiled BUG_ON(),
adding debug code right into the heart of critical routines like
interrupt handlers.

   text	   data	    bss	    dec	    hex
1296480	  19944	   2272	1318696	 141f28	before (lockdep disabled)
1295984	  19944	   2272	1318200	 141d38	after

1336261	  21139	   3208	1360608	 14c2e0	before (lockdep enabled)
1339920	  21139	   3208	1364267	 14d12b	after

Small saving for release; hopefully more instructive in debug.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170302132801.599-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

Everytime we take the fence->lock (aka request->lock), we must do so
with irqs disabled since it may be used from within an hardirq context.
As sometimes we are taking the lock in a nested manner, assert that the
caller did disable the irqs for us.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170302115130.28434-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>