In order to allow the mock breadcrumbs tests to run without device irqs
being enabled, move the intel_irqs_enabled() assert deeper to just
before we commit to enabling the HW irq.

v2: Add a FIXME explaining that placing the assertion so deep is not
ideal, but a compromise for mock breadcrumbs.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Matthew Auld <matthew.william.auld@gmail.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171107102003.1802-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

We will disarm the breadcrumb interrupt if we see a user interrupt
whilst no one is waiting. This may race with the call to
intel_engine_disarm_breadcrumbs() triggering an assert that we aren't
trying to do the same job twice. Prevent this by checking that the irq
is still armed after flushing the interrupt (for the irq spinlock).

Fixes: bcbd5c33 ("drm/i915/guc: Always enable the breadcrumbs irq")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171031122235.1395-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

The execlists emulation on top of the GuC (used for scheduling and
preemption) depends on the MI_USER_INTERRUPT for its notifications and
tasklet action. As we always employ the irq, there is no advantage in
ever disabling it while we are using the GuC, so allow us to arm the
breadcrumb irq when enabling GuC submission and disarm upon disabling.
The impact should be lessened by the delayed irq disabling we do (we
only disable after receiving an interrupt for which no one was wanting),
but allowing guc to explicitly manage the irq in relation to itself is
simpler and prevents an issue with losing an interrupt for preemption
as it is not coupled to an active request.

Internally, we add a reference counter (breadcrumbs.irq_enabled) as a
simple mechanism to allow GuC to keep the breadcrumb irq enabled. To
improve upon always enabling the irq while guc is selected, we need
to hook into the parking facility of intel_engines so that we only enable
the breadcrumbs while the GT is active (one step better would be to
individually park/unpark each engine).

In effect, this means that we keep the breadcrumb irq always enabled for
the entire duration the guc is busy, whereas before we would try to
switch it off whenever we idled for more than interrupt with no
associated waiters. The difference *should* be negligible in practice!

v2: Stop abusing fence signaling (and its auxiliary data structures) to
enable the breadcrumbs irqs.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Michał Winiarski <michal.winiarski@intel.com>,
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>,
Link: https://patchwork.freedesktop.org/patch/msgid/20171025143943.7661-3-chris@chris-wilson.co.uk

In preparation for unconditionally passing the struct timer_list pointer to
all timer callbacks, switch to using the new timer_setup() and from_timer()
to pass the timer pointer explicitly.

Cc: Daniel Vetter <daniel.vetter@intel.com>
Cc: Jani Nikula <jani.nikula@linux.intel.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Oscar Mateo <oscar.mateo@intel.com>
Cc: intel-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Signed-off-by: NKees Cook <keescook@chromium.org>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Signed-off-by: NJani Nikula <jani.nikula@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171017065304.3358-1-joonas.lahtinen@linux.intel.com

Use the %pS printk format for printing symbols from direct addresses.
This is important for the ia64, ppc64 and parisc64 architectures, while on
other architectures there is no difference between %pS and %pF.
Fix it for consistency across the kernel.
Signed-off-by: NHelge Deller <deller@gmx.de>
Cc: Jani Nikula <jani.nikula@linux.intel.com>
Cc: David Airlie <airlied@linux.ie>
Cc: intel-gfx@lists.freedesktop.org
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/1504729681-3504-6-git-send-email-deller@gmx.de

Enabling the interrupt for the signaler takes a finite amount of time (a
few microseconds) during which it is possible for the request to
complete. Check afterwards and skip adding the request to the signal
rbtree if it complete.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170608111405.16466-3-chris@chris-wilson.co.uk

The important condition that we need to check after enabling the
interrupt for signaling is whether the request completed in the process
(and so we missed that interrupt). A large cost in enabling the
signaling (rather than waiters) is in waking up the auxiliary signaling
thread, but we only need to do so to catch that missed interrupt. If we
know we didn't miss any interrupts (because we didn't arm the interrupt)
then we can skip waking the auxiliary thread.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170608111405.16466-2-chris@chris-wilson.co.uk

If the engine is continually completing nops, we can saturate the
signaler and keep it working indefinitely. This angers the NMI watchdog!

A good example is to disable semaphores on snb and run igt/gem_exec_nop -
the parallel, multi-engine workloads are more than sufficient to hog the
CPU, preventing the system from even processing ICMP echo replies.

v2: Tvrtko dug into cond_resched() on x86 and found that it only
depended upon preempt_count and not tif_need_resched() - which means
that we would always call schedule() at that point.

Fixes: c81d4613 ("drm/i915: Convert trace-irq to the breadcrumb waiter")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170404120531.10737-1-chris@chris-wilson.co.ukReviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
(cherry picked from commit a7980a64)
Signed-off-by: NJani Nikula <jani.nikula@intel.com>

If the signal to park arrives before we sleep, then we need to check
kthread_should_park() before sleeping to avoid missing the signal.
Otherwise, if the signal arrives whilst we are processing completed
requests, we will reset the current->state back to TASK_INTERRUPTIBLE
and so miss the wakeup.

Fixes: fe3288b5 ("drm/i915: Park the breadcrumbs signaler across a GPU reset")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170403105124.8969-1-chris@chris-wilson.co.ukReviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
(cherry picked from commit b1becb88)
Signed-off-by: NJani Nikula <jani.nikula@intel.com>

If we are enabling the breadcrumbs signaling prior to submitting the
request, we know that we cannot have missed the interrupt and can
therefore skip immediately waking the signaler to check.

This reduces a significant chunk of the __i915_gem_request_submit()
overhead for inter-engine synchronisation, for example in gem_exec_whisper.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170426080659.28771-1-chris@chris-wilson.co.ukReviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>

Knowing the neighbouring seqno (current on hw, last submitted to hw)
provide some useful breadcrumbs to the debug log.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: http://patchwork.freedesktop.org/patch/msgid/20170423170619.7156-4-chris@chris-wilson.co.ukReviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>

If the engine is continually completing nops, we can saturate the
signaler and keep it working indefinitely. This angers the NMI watchdog!

A good example is to disable semaphores on snb and run igt/gem_exec_nop -
the parallel, multi-engine workloads are more than sufficient to hog the
CPU, preventing the system from even processing ICMP echo replies.

v2: Tvrtko dug into cond_resched() on x86 and found that it only
depended upon preempt_count and not tif_need_resched() - which means
that we would always call schedule() at that point.

Fixes: c81d4613 ("drm/i915: Convert trace-irq to the breadcrumb waiter")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170404120531.10737-1-chris@chris-wilson.co.ukReviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>

If the signal to park arrives before we sleep, then we need to check
kthread_should_park() before sleeping to avoid missing the signal.
Otherwise, if the signal arrives whilst we are processing completed
requests, we will reset the current->state back to TASK_INTERRUPTIBLE
and so miss the wakeup.

Fixes: fe3288b5 ("drm/i915: Park the breadcrumbs signaler across a GPU reset")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170403105124.8969-1-chris@chris-wilson.co.ukReviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>

intel_engine_wakeup() is called by nop_request_submit() which is
installed to handle third party fences completed from within irq
context. As such, it needs the full irqsave/irqrestore and not the
partial spin_irq_lock handling.

[18942.714467] =================================
[18942.719076] [ INFO: inconsistent lock state ]
[18942.723522] 4.11.0-rc2-CI-CI_DRM_2368+ #1 Tainted: G     U  W
[18942.729970] ---------------------------------
[18942.734466] inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
[18942.740594] gem_eio/1275 [HC0[0]:SC0[0]:HE1:SE1] takes:
[18942.745932]  (&(&fence->lock)->rlock){+.?...}, at: [<ffffffff815ec100>] dma_fence_signal+0x100/0x
230
[18942.755331] {IN-SOFTIRQ-W} state was registered at:
[18942.760356]   __lock_acquire+0x5d0/0x1bb0
[18942.764444]   lock_acquire+0xc9/0x220
[18942.768196]   _raw_spin_lock_irqsave+0x41/0x60
[18942.772747]   dma_fence_signal+0x100/0x230
[18942.776927]   vgem_fence_timeout+0x9/0x10 [vgem]
[18942.781701]   call_timer_fn+0x92/0x380
[18942.785557]   expire_timers+0x150/0x1f0
[18942.789491]   run_timer_softirq+0x7c/0x160
[18942.793705]   __do_softirq+0x116/0x4c0
[18942.797560]   irq_exit+0xa9/0xc0
[18942.800873]   smp_apic_timer_interrupt+0x38/0x50
[18942.805611]   apic_timer_interrupt+0x90/0xa0
[18942.810008]   cpuidle_enter_state+0x135/0x380
[18942.814503]   cpuidle_enter+0x12/0x20
[18942.818250]   call_cpuidle+0x1e/0x40
[18942.821906]   do_idle+0x17e/0x1f0
[18942.825333]   cpu_startup_entry+0x18/0x20
[18942.829463]   rest_init+0x127/0x130
[18942.833025]   start_kernel+0x3f1/0x3fe
[18942.836908]   x86_64_start_reservations+0x2a/0x2c
[18942.841733]   x86_64_start_kernel+0x173/0x186
[18942.846234]   verify_cpu+0x0/0xfc
[18942.849604] irq event stamp: 30568
[18942.853140] hardirqs last  enabled at (30567): [<ffffffff8110b81f>] ktime_get+0xef/0x120
[18942.861468] hardirqs last disabled at (30568): [<ffffffff81876377>] _raw_spin_lock_irqsave+0x17/0
x60
[18942.870812] softirqs last  enabled at (30462): [<ffffffff81085cd9>] __do_softirq+0x1d9/0x4c0
[18942.879443] softirqs last disabled at (30439): [<ffffffff81086139>] irq_exit+0xa9/0xc0
[18942.887616]
[18942.887616] other info that might help us debug this:
[18942.894279]  Possible unsafe locking scenario:
[18942.894279]
[18942.900336]        CPU0
[18942.902851]        ----
[18942.905362]   lock(&(&fence->lock)->rlock);
[18942.909647]   <Interrupt>
[18942.912330]     lock(&(&fence->lock)->rlock);
[18942.916821]
[18942.916821]  *** DEADLOCK ***
[18942.916821]
[18942.922862] 1 lock held by gem_eio/1275:
[18942.926859]  #0:  (&(&fence->lock)->rlock){+.?...}, at: [<ffffffff815ec100>] dma_fence_signal+0x1
00/0x230
[18942.936651]
[18942.936651] stack backtrace:
[18942.941142] CPU: 3 PID: 1275 Comm: gem_eio Tainted: G     U  W       4.11.0-rc2-CI-CI_DRM_2368+ #
1
[18942.950367] Hardware name: Gigabyte Technology Co., Ltd. Z170X-UD5/Z170X-UD5-CF, BIOS F21 01/06/2
017
[18942.959756] Call Trace:
[18942.962244]  dump_stack+0x67/0x92
[18942.965626]  print_usage_bug.part.23+0x259/0x268
[18942.970362]  mark_lock+0x12c/0x6f0
[18942.973851]  ? check_usage_forwards+0x130/0x130
[18942.978487]  mark_held_locks+0x6f/0xa0
[18942.982329]  ? _raw_spin_unlock_irq+0x27/0x50
[18942.986797]  trace_hardirqs_on_caller+0x150/0x200
[18942.991599]  trace_hardirqs_on+0xd/0x10
[18942.995515]  _raw_spin_unlock_irq+0x27/0x50
[18942.999796]  intel_engine_wakeup+0x26/0x30 [i915]
[18943.004670]  intel_engine_init_global_seqno+0x131/0x1a0 [i915]
[18943.010745]  nop_submit_request+0x2e/0x40 [i915]
[18943.015476]  submit_notify+0x3f/0x5c [i915]
[18943.019763]  __i915_sw_fence_complete+0x176/0x220 [i915]
[18943.025234]  ? try_to_del_timer_sync+0x4d/0x60
[18943.029825]  i915_sw_fence_complete+0x25/0x40 [i915]
[18943.034887]  dma_i915_sw_fence_wake+0x26/0x60 [i915]
[18943.039959]  dma_fence_signal+0x146/0x230
[18943.044109]  vgem_fence_signal_ioctl+0x6c/0xc0 [vgem]
[18943.049275]  drm_ioctl+0x200/0x450
[18943.052758]  ? vgem_fence_attach_ioctl+0x270/0x270 [vgem]
[18943.058334]  do_vfs_ioctl+0x90/0x6e0
[18943.061991]  ? entry_SYSCALL_64_fastpath+0x5/0xb1
[18943.066843]  ? __this_cpu_preempt_check+0x13/0x20
[18943.071643]  ? trace_hardirqs_on_caller+0xe7/0x200
[18943.076532]  SyS_ioctl+0x3c/0x70
[18943.079842]  entry_SYSCALL_64_fastpath+0x1c/0xb1
[18943.084558] RIP: 0033:0x7f0dfcc14357
[18943.088240] RSP: 002b:00007ffeb4628da8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[18943.095996] RAX: ffffffffffffffda RBX: ffffffff8147eb93 RCX: 00007f0dfcc14357
[18943.103311] RDX: 00007ffeb4628de0 RSI: 0000000040086442 RDI: 0000000000000005
[18943.110574] RBP: ffffc9000176ff88 R08: 0000000000000004 R09: 0000000000000000
[18943.117845] R10: 0000000000000029 R11: 0000000000000246 R12: 0000000000000001
[18943.125168] R13: 0000000000000005 R14: 0000000040086442 R15: 0000000000000000
[18943.132520]  ? __this_cpu_preempt_check+0x13/0x20

Fixes: cdc3a453 ("drm/i915: No need to save/restore irq status in intel_engine_wakeup")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170320143133.1507-1-chris@chris-wilson.co.ukReviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>

Tvrtko spotted a stale reference to b->lock (now b->rb_lock) so review
the comments and try to improve them in passing.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170315222259.1469-1-chris@chris-wilson.co.ukReviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>

We need to ensure that we always serialize updates to the bottom-half
using the breadcrumbs.irq_lock so that we don't race with a concurrent
interrupt handler. This is most important just prior to leaving the
waiter (when the intel_wait will be overwritten), so make sure we are
not the current bottom-half when skipping the irq locks.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170315210726.12095-4-chris@chris-wilson.co.uk

Before walking the rbtree of waiters (marking them as complete and waking
them), decouple the interrupt handler. This prevents a race between the
missed waiter waking up and removing its intel_wait (which skips
checking the lock) and the interrupt handler dereferencing the
intel_wait. (Though we do not expect to encounter waiters during idle!)

Fixes: e1c0c91b ("drm/i915: Wake up all waiters before idling")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170315210726.12095-3-chris@chris-wilson.co.uk

When adding a new request to the breadcrumb rbtree, we mark all those
requests inside the rbtree that are already completed as complete. This
wakes those waiters up and allows them to skip the spinlock before
returning to userspace. If one of those is the current bottom-half and
allocated its intel_wait on the stack, it may then overwrite the
b->irq_wait upon exiting i915_wait_request() just as the interrupt handler
dereferences it.

Fixes: 56299fb7 ("drm/i915: Signal first fence from irq handler if complete")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170315210726.12095-2-chris@chris-wilson.co.uk

Since commit 9b6586ae ("drm/i915: Keep a global seqno per-engine")
converted intel_breadcrumbs_busy() to reporting a single boolean, we
need only compute a boolean internally (and not needlessly compute the
flag).
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170315210726.12095-1-chris@chris-wilson.co.uk

It is called from either the process or timer context so it is
correct to always disable interrupts.
Signed-off-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: http://patchwork.freedesktop.org/patch/msgid/20170306150321.29024-1-tvrtko.ursulin@linux.intel.com

Timer callback is a known context so it is correct to always
disable interrupts.
Signed-off-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>

When we idle, we wakeup the first waiter (checking to see if it missed
an earlier wakeup) and disarm the breadcrumbs. However, we now assert
that there are no waiter when the interrupt is disabled, triggering an
assert if there were multiple waiters when we idled.

[  420.842275] invalid opcode: 0000 [#1] PREEMPT SMP
[  420.842285] Modules linked in: vgem snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_intel snd_hda_codec snd_hwdep mei_me snd_hda_core mei snd_pcm lpc_ich i915 r8169 mii prime_numbers
[  420.842357] CPU: 4 PID: 8714 Comm: kms_pipe_crc_ba Tainted: G     U  W       4.10.0-CI-CI_DRM_2280+ #1
[  420.842377] Hardware name: Hewlett-Packard HP Pro 3500 Series/2ABF, BIOS 8.11 10/24/2012
[  420.842395] task: ffff880117ddce40 task.stack: ffffc90001114000
[  420.842439] RIP: 0010:__intel_engine_remove_wait+0x1f4/0x200 [i915]
[  420.842454] RSP: 0018:ffffc90001117b18 EFLAGS: 00010046
[  420.842467] RAX: 0000000000000000 RBX: ffff88010c25c2a8 RCX: 0000000000000001
[  420.842481] RDX: 0000000000000001 RSI: 00000000ffffffff RDI: ffffc90001117c50
[  420.842495] RBP: ffffc90001117b58 R08: 0000000011e52352 R09: c4d16acc00000000
[  420.842511] R10: ffffffff82789eb0 R11: ffff880117ddce40 R12: ffffc90001117c50
[  420.842525] R13: ffffc90001117c50 R14: 0000000000000078 R15: 0000000000000000
[  420.842540] FS:  00007fe47dda0a40(0000) GS:ffff88011fb00000(0000) knlGS:0000000000000000
[  420.842559] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  420.842571] CR2: 00007fd6c0a2cec4 CR3: 000000010a5e5000 CR4: 00000000001406e0
[  420.842586] Call Trace:
[  420.842595]  ? do_raw_spin_lock+0xad/0xb0
[  420.842635]  intel_engine_remove_wait.part.3+0x26/0x40 [i915]
[  420.842678]  intel_engine_remove_wait+0xe/0x20 [i915]
[  420.842721]  i915_wait_request+0x4f0/0x8c0 [i915]
[  420.842736]  ? wake_up_q+0x70/0x70
[  420.842747]  ? wake_up_q+0x70/0x70
[  420.842787]  i915_gem_object_wait_fence+0x7d/0x1a0 [i915]
[  420.842829]  i915_gem_object_wait+0x30d/0x520 [i915]
[  420.842842]  ? __this_cpu_preempt_check+0x13/0x20
[  420.842884]  i915_gem_wait_ioctl+0x12e/0x2e0 [i915]
[  420.842924]  ? i915_gem_wait_ioctl+0x22/0x2e0 [i915]
[  420.842939]  drm_ioctl+0x200/0x450
[  420.842976]  ? i915_gem_set_wedged+0x90/0x90 [i915]
[  420.842993]  do_vfs_ioctl+0x90/0x6e0
[  420.843003]  ? entry_SYSCALL_64_fastpath+0x5/0xb1
[  420.843017]  ? __this_cpu_preempt_check+0x13/0x20
[  420.843030]  ? trace_hardirqs_on_caller+0xe7/0x200
[  420.843042]  SyS_ioctl+0x3c/0x70
[  420.843054]  entry_SYSCALL_64_fastpath+0x1c/0xb1
[  420.843065] RIP: 0033:0x7fe47c4b9357
[  420.843075] RSP: 002b:00007ffc3c0633c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  420.843094] RAX: ffffffffffffffda RBX: ffffffff81482393 RCX: 00007fe47c4b9357
[  420.843109] RDX: 00007ffc3c063400 RSI: 00000000c010646c RDI: 0000000000000004
[  420.843123] RBP: ffffc90001117f88 R08: 0000000000000008 R09: 0000000000000000
[  420.843137] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[  420.843151] R13: 0000000000000004 R14: 00000000c010646c R15: 0000000000000000
[  420.843168]  ? __this_cpu_preempt_check+0x13/0x20
[  420.843180] Code: 81 48 c7 c1 40 6a 16 a0 48 c7 c2 47 29 15 a0 be 17 01 00 00 48 c7 c7 10 6a 16 a0 e8 c7 ea fe e0 e9 5d ff ff ff 0f 0b 0f 0b 0f 0b <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 67 41 7e e1
[  420.843325] RIP: __intel_engine_remove_wait+0x1f4/0x200 [i915] RSP: ffffc90001117b18

Fixes: b66255f0 ("drm/i915: Refactor wakeup of the next breadcrumb waiter")
Fixes: 67b807a8 ("drm/i915: Delay disabling the user interrupt for breadcrumbs")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170306092916.11623-2-chris@chris-wilson.co.ukReviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>

As we now take the breadcrumbs spinlock within the interrupt handler, we
wish to minimise its hold time. During the interrupt we do not care
about the state of the full rbtree, only that of the first element, so
we can guard that with a separate lock.

v2: Rename first_wait to irq_wait to make it clearer that it is guarded
by irq_lock.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170303190824.1330-1-chris@chris-wilson.co.uk

Refactor the common task of updating the first_waiter, serialised with
the interrupt handler. When we update the first_waiter, we also need to
wakeup the new bottom-half in order to complete the actions that we may
have delegated to it (such as checking the irq-seqno coherency or waking
up other lower priority concurrent waiters).
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NMika Kuoppala <mika.kuoppala@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170303171422.4735-1-chris@chris-wilson.co.uk

assert_spin_locked() becomes an unconditionally compiled BUG_ON(),
adding debug code right into the heart of critical routines like
interrupt handlers.

   text	   data	    bss	    dec	    hex
1296480	  19944	   2272	1318696	 141f28	before (lockdep disabled)
1295984	  19944	   2272	1318200	 141d38	after

1336261	  21139	   3208	1360608	 14c2e0	before (lockdep enabled)
1339920	  21139	   3208	1364267	 14d12b	after

Small saving for release; hopefully more instructive in debug.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170302132801.599-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

Everytime we take the fence->lock (aka request->lock), we must do so
with irqs disabled since it may be used from within an hardirq context.
As sometimes we are taking the lock in a nested manner, assert that the
caller did disable the irqs for us.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170302115130.28434-1-chris@chris-wilson.co.ukReviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

We are going to move scheduler ABI details to <uapi/linux/sched/types.h>,
which will be used from a number of .c files.

Create empty placeholder header that maps to <linux/types.h>.

Include the new header in the files that are going to need it.
Acked-by: NLinus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: NIngo Molnar <mingo@kernel.org>

Move the setting of gpu_error->missed_irq_ring bit to a common function
so that we can get the debug logging for either path.

v2: Add %pF caller
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170228085018.3225-1-chris@chris-wilson.co.uk

A significant cost in setting up a wait is the overhead of enabling the
interrupt. As we disable the interrupt whenever the queue of waiters is
empty, if we are frequently waiting on alternating batches, we end up
re-enabling the interrupt on a frequent basis. We do want to disable the
interrupt during normal operations as under high load it may add several
thousand interrupts/s - we have been known in the past to occupy whole
cores with our interrupt handler after accidentally leaving user
interrupts enabled. As a compromise, leave the interrupt enabled until
the next IRQ, or the system is idle. This gives a small window for a
waiter to keep the interrupt active and not be delayed by having to
re-enable the interrupt.

v2: Restore hangcheck/missed-irq detection for continuations
v3: Be more careful restoring the hangcheck timer after reset
v4: Be more careful restoring the fake irq after reset (if required!)
v5: Redo changes to intel_engine_wakeup()
v6: Factor out __intel_engine_wakeup()
v7: Improve commentary for declaring a missed wakeup
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170227205850.2828-4-chris@chris-wilson.co.uk

By deferring hangcheck to the fake breadcrumb interrupt, we can simply
the enabling procedure slightly - as by enabling the fake, we then
enable the hangcheck. By always enabling the hangcheck from each fake
interrupt (it will be a no-op for an already queued hangcheck), it will
make restoring the breadcrumbs after a reset simpler in the next patch.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170227205850.2828-3-chris@chris-wilson.co.uk

As execlists and other non-semaphore multi-engine devices coordinate
between engines using interrupts, we can shave off a few 10s of
microsecond of scheduling latency by doing the fence signaling from the
interrupt as opposed to a RT kthread. (Realistically the delay adds
about 1% to an individual cross-engine workload.) We only signal the
first fence in order to limit the amount of work we move into the
interrupt handler. We also have to remember that our breadcrumbs may be
unordered with respect to the interrupt and so we still require the
waiter process to perform some heavyweight coherency fixups, as well as
traversing the tree of waiters.

v2: No need for early exit in irq handler - it breaks the flow between
patches and prevents the tracepoint
v3: Restore rcu hold across irq signaling of request
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170227205850.2828-2-chris@chris-wilson.co.uk

The two users of the return value from intel_engine_wakeup() are
expecting different results. In the breadcrumbs hangcheck, we are using
it to determine whether wake_up_process() detected the waiter was
currently running (and if so we presume that it hasn't yet missed the
interrupt). However, in the fake_irq path, we are using the return value
as a check as to whether there are any waiters, and so we may
incorrectly stop the fake-irq if that waiter was currently running.

To handle the two different needs, return both bits of information! We
uninline it from the irq path in preparation for the next patch which
makes the irq hotpath special and relegates intel_engine_wakeup() to the
slow fixup paths.

v2: s/ret/result/
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170227205850.2828-1-chris@chris-wilson.co.uk

After the request is cancelled, we then need to remove it from the
global execution timeline and return it to the context timeline, the
inverse of submit_request().

v2: Move manipulation of struct intel_wait to helpers
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170223074422.4125-12-chris@chris-wilson.co.ukReviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>

If we preempt a request and remove it from the execution queue, we need
to undo its global seqno and restart any waiters.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170223074422.4125-11-chris@chris-wilson.co.uk

The plan in the near-future is to allow requests to be removed from the
signaler. We can no longer then rely on holding a reference to the
request for the duration it is in the signaling tree, and instead must
obtain a reference to the request for the current operation using RCU.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170223074422.4125-10-chris@chris-wilson.co.uk

A request is assigned a global seqno only when it is on the hardware
execution queue. The global seqno can be used to maintain a list of
requests on the same engine in retirement order, for example for
constructing a priority queue for waiting. Prior to its execution, or
if it is subsequently removed in the event of preemption, its global
seqno is zero. As both insertion and removal from the execution queue
may operate in IRQ context, it is not guarded by the usual struct_mutex
BKL. Instead those relying on the global seqno must be prepared for its
value to change between reads. Only when the request is complete can
the global seqno be stable (due to the memory barriers on submitting
the commands to the hardware to write the breadcrumb, if the HWS shows
that it has passed the global seqno and the global seqno is unchanged
after the read, it is indeed complete).
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170223074422.4125-9-chris@chris-wilson.co.uk

Replace the global device seqno with one for each engine, and account
for in-flight seqno on each separately. This is consistent with
dma-fence as each timeline has separate fence-contexts for each engine
and a seqno is only ordered within a fence-context (i.e.  seqno do not
need to be ordered wrt to other engines, just ordered within a single
engine). This is required to enable request rewinding for preemption on
individual engines (we have to rewind the global seqno to avoid
overflow, and we do not have to rewind all engines just to preempt one.)

v2: Rename active_seqno to inflight_seqnos to more clearly indicate that
it is a counter and not equivalent to the existing seqno. Update
functions that operated on active_seqno similarly.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170223074422.4125-3-chris@chris-wilson.co.uk

As a backup to waiting on a user-interrupt from the GPU, we use a heavy
and frequent timer to wake up the waiting process should we detect an
inconsistency whilst waiting. After seeing a "missed interrupt", the
next time we wait, we restart the heavy timer. This patch is more
reluctant to restart the timer and will only do so if we have not see any
interrupts since when we started the fake irq timer. If we are seeing
interrupts, then the waiters are being woken normally and we had an
incoherency that caused to miss last time - that is unlikely to reoccur
and so taking the risk of stalling again seems pragmatic.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170217151304.16665-5-chris@chris-wilson.co.uk

If the waiter was currently running, assume it hasn't had a chance
to process the pending interrupt (e.g, low priority task on a loaded
system) and wait until it sleeps before declaring a missed interrupt.

References: https://bugs.freedesktop.org/show_bug.cgi?id=99816Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170217151304.16665-4-chris@chris-wilson.co.uk

When the timer expires for checking on interrupt processing, check to
see if any interrupts arrived within the last time period. If real
interrupts are still being delivered, we can be reassured that we
haven't missed the final interrupt as the waiter will still be woken.
Only once all activity ceases, do we have to worry about the waiter
never being woken and so need to install a timer to kick the waiter for
a slow arrival of a seqno.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170217151304.16665-2-chris@chris-wilson.co.uk