Since the breadcrumb enabling/cancelling itself is serialised by the
breadcrumbs.irq_lock, with a bit of care we can remove the outer
serialisation with i915_request.lock for concurrent
dma_fence_enable_signaling(). This has the important side-effect of
eliminating the nested i915_request.lock within request submission.

The challenge in serialisation is around the unsubmission where we take
an active request that wants a breadcrumb on the signaling engine and
put it to sleep. We do not want a concurrent
dma_fence_enable_signaling() to attach a breadcrumb as we unsubmit, so
we must mark the request as no longer active before serialising with the
concurrent enable-signaling.

On retire, we serialise with the concurrent enable-signaling, but
instead of clearing ACTIVE, we mark it as SIGNALED.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200731154834.8378-1-chris@chris-wilson.co.ukSigned-off-by: NRodrigo Vivi <rodrigo.vivi@intel.com>
[Joonas: Rebased and reordered into drm-intel-gt-next branch]
Signed-off-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

Preempt-to-busy introduces various fascinating complications in that the
requests may complete as we are unsubmitting them from HW. As they may
then signal after unsubmission, we may find ourselves having to cleanup
the signaling request from within the signaling callback. This causes us
to recurse onto the same i915_request.lock.

However, if the request is already signaled (as it will be before we
enter the signal callbacks), we know we can skip the signaling of that
request during submission, neatly evading the spinlock recursion.

unsubmit(ve.rq0) # timeslice expiration or other preemption
 -> virtual_submit_request(ve.rq0)
dma_fence_signal(ve.rq0) # request completed before preemption ack
 -> submit_notify(ve.rq1)
   -> virtual_submit_request(ve.rq1) # sees that we have completed ve.rq0
      -> __i915_request_submit(ve.rq0)

[  264.210142] BUG: spinlock recursion on CPU#2, sample_multi_tr/2093
[  264.210150]  lock: 0xffff9efd6ac55080, .magic: dead4ead, .owner: sample_multi_tr/2093, .owner_cpu: 2
[  264.210155] CPU: 2 PID: 2093 Comm: sample_multi_tr Tainted: G     U
[  264.210158] Hardware name: Intel Corporation CoffeeLake Client Platform/CoffeeLake S UDIMM RVP, BIOS CNLSFWR1.R00.X212.B01.1909060036 09/06/2019
[  264.210160] Call Trace:
[  264.210167]  dump_stack+0x98/0xda
[  264.210174]  spin_dump.cold+0x24/0x3c
[  264.210178]  do_raw_spin_lock+0x9a/0xd0
[  264.210184]  _raw_spin_lock_nested+0x6a/0x70
[  264.210314]  __i915_request_submit+0x10a/0x3c0 [i915]
[  264.210415]  virtual_submit_request+0x9b/0x380 [i915]
[  264.210516]  submit_notify+0xaf/0x14c [i915]
[  264.210602]  __i915_sw_fence_complete+0x8a/0x230 [i915]
[  264.210692]  i915_sw_fence_complete+0x2d/0x40 [i915]
[  264.210762]  __dma_i915_sw_fence_wake+0x19/0x30 [i915]
[  264.210767]  dma_fence_signal_locked+0xb1/0x1c0
[  264.210772]  dma_fence_signal+0x29/0x50
[  264.210871]  i915_request_wait+0x5cb/0x830 [i915]
[  264.210876]  ? dma_resv_get_fences_rcu+0x294/0x5d0
[  264.210974]  i915_gem_object_wait_fence+0x2f/0x40 [i915]
[  264.211084]  i915_gem_object_wait+0xce/0x400 [i915]
[  264.211178]  i915_gem_wait_ioctl+0xff/0x290 [i915]

Fixes: 22b7a426 ("drm/i915/execlists: Preempt-to-busy")
References: 6d06779e ("drm/i915: Load balancing across a virtual engine")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: "Nayana, Venkata Ramana" <venkata.ramana.nayana@intel.com>
Cc: <stable@vger.kernel.org> # v5.4+
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200713141636.29326-1-chris@chris-wilson.co.uk

The second try at staging the transfer of the breadcrumb. In part one,
we realised we could not simply move to the second engine as we were
only holding the breadcrumb lock on the first. So in commit 6c81e21a
("drm/i915/gt: Stage the transfer of the virtual breadcrumb"), we
removed it from the first engine and marked up this request to reattach
the signaling on the new engine. However, this failed to take into
account that we only attach the breadcrumb if the new request is added
at the start of the queue, which if we are transferring, it is because
we know there to be a request to be signaled (and hence we would not be
attached).

In this attempt, we try to transfer the completed requests to the
irq_worker on its rq->engine->breadcrumbs. This preserves the coupling
between the rq and its breadcrumbs, so that
i915_request_cancel_breadcrumb() does not attempt to manipulate the list
under the wrong lock.

v2: Code sharing is fun.

Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/1862
Fixes: 6c81e21a ("drm/i915/gt: Stage the transfer of the virtual breadcrumb")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200513074809.18194-1-chris@chris-wilson.co.uk

We control b->irq_enabled inside the b->irq_lock, but we check before
entering the spinlock whether or not the interrupt is currently
unmasked.

[ 1511.735208] BUG: KCSAN: data-race in __intel_breadcrumbs_disarm_irq [i915] / intel_engine_disarm_breadcrumbs [i915]
[ 1511.735231]
[ 1511.735242] write to 0xffff8881f75fc214 of 1 bytes by interrupt on cpu 2:
[ 1511.735440]  __intel_breadcrumbs_disarm_irq+0x4b/0x160 [i915]
[ 1511.735635]  signal_irq_work+0x337/0x710 [i915]
[ 1511.735652]  irq_work_run_list+0xd7/0x110
[ 1511.735666]  irq_work_run+0x1d/0x50
[ 1511.735681]  smp_irq_work_interrupt+0x21/0x30
[ 1511.735701]  irq_work_interrupt+0xf/0x20
[ 1511.735722]  __do_softirq+0x6f/0x206
[ 1511.735736]  irq_exit+0xcd/0xe0
[ 1511.735756]  do_IRQ+0x44/0xc0
[ 1511.735773]  ret_from_intr+0x0/0x1c
[ 1511.735787]  schedule+0x0/0xb0
[ 1511.735803]  worker_thread+0x194/0x670
[ 1511.735823]  kthread+0x19a/0x1e0
[ 1511.735837]  ret_from_fork+0x1f/0x30
[ 1511.735848]
[ 1511.735867] read to 0xffff8881f75fc214 of 1 bytes by task 432 on cpu 1:
[ 1511.736068]  intel_engine_disarm_breadcrumbs+0x22/0x80 [i915]
[ 1511.736263]  __engine_park+0x107/0x5d0 [i915]
[ 1511.736453]  ____intel_wakeref_put_last+0x44/0x90 [i915]
[ 1511.736648]  __intel_wakeref_put_last+0x5a/0x70 [i915]
[ 1511.736842]  intel_context_exit_engine+0xf2/0x100 [i915]
[ 1511.737044]  i915_request_retire+0x6b2/0x770 [i915]
[ 1511.737244]  retire_requests+0x7a/0xd0 [i915]
[ 1511.737438]  intel_gt_retire_requests_timeout+0x3a7/0x6f0 [i915]
[ 1511.737633]  i915_drop_caches_set+0x1e7/0x260 [i915]
[ 1511.737650]  simple_attr_write+0xfa/0x110
[ 1511.737665]  full_proxy_write+0x94/0xc0
[ 1511.737679]  __vfs_write+0x4b/0x90
[ 1511.737697]  vfs_write+0xfc/0x280
[ 1511.737718]  ksys_write+0x78/0x100
[ 1511.737732]  __x64_sys_write+0x44/0x60
[ 1511.737751]  do_syscall_64+0x6e/0x2c0
[ 1511.737769]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200408092916.5355-1-chris@chris-wilson.co.uk

Virtual engines are fleeting. They carry a reference count and may be freed
when their last request is retired. This makes them unsuitable for the
task of housing engine->retire.work so assert that it is not used.

Tvrtko tracked down an instance where we did indeed violate this rule.
In virtual_submit_request, we flush a completed request directly with
__i915_request_submit and this causes us to queue that request on the
veng's breadcrumb list and signal it. Leading us down a path where we
should not attach the retire.
Reported-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Fixes: dc93c9b6 ("drm/i915/gt: Schedule request retirement when signaler idles")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200206204915.2636606-1-chris@chris-wilson.co.uk
(cherry picked from commit f91d8156)
Signed-off-by: NJani Nikula <jani.nikula@intel.com>

Virtual engines are fleeting. They carry a reference count and may be freed
when their last request is retired. This makes them unsuitable for the
task of housing engine->retire.work so assert that it is not used.

Tvrtko tracked down an instance where we did indeed violate this rule.
In virtual_submit_request, we flush a completed request directly with
__i915_request_submit and this causes us to queue that request on the
veng's breadcrumb list and signal it. Leading us down a path where we
should not attach the retire.
Reported-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Fixes: dc93c9b6 ("drm/i915/gt: Schedule request retirement when signaler idles")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200206204915.2636606-1-chris@chris-wilson.co.uk

Keep the intel_context as being the primary state for i915_request, with
the GEM context a backpointer from the low level state for the rarer
cases we need client information. Our goal is to remove such references
to clients from the backend, and leave the HW submission agnostic to
client interfaces and self-contained.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Andi Shyti <andi.shyti@intel.com>
Reviewed-by: NAndi Shyti <andi.shyti@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191220101230.256839-1-chris@chris-wilson.co.uk

Avoid adding the retire workers to the virtual engine so that we don't
end up in the unenviable situation of trying to free the virtual engine
while its worker remains active.

Fixes: dc93c9b6 ("drm/i915/gt: Schedule request retirement when signaler idles")
Closes: https://gitlab.freedesktop.org/drm/intel/issues/867Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Acked-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191219221344.161523-1-chris@chris-wilson.co.uk

Very similar to commit 4f88f874 ("drm/i915/gt: Schedule request
retirement when timeline idles"), but this time instead of coupling into
the execlists CS event interrupt, we couple into the breadcrumb
interrupt and queue a timeline's retirement when the last signaler is
completed. This should allow us to more rapidly park ringbuffer
submission, and so help reduce power consumption on older systems.

v2: Fixup intel_engine_add_retire() to handle concurrent callers

References: 4f88f874 ("drm/i915/gt: Schedule request retirement when timeline idles")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191219124353.8607-1-chris@chris-wilson.co.uk

Only signal the breadcrumbs from inside the irq_work, simplifying our
interface and calling conventions. The micro-optimisation here is that
by always using the irq_work interface, we know we are always inside an
irq-off critical section for the breadcrumb signaling and can ellide
save/restore of the irq flags.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191217095642.3124521-7-chris@chris-wilson.co.uk

Take a wakeref on the intel_gt specifically for the enabled breadcrumb
interrupt so that we can safely process the mmio. If the intel_gt is
already asleep by the time we try and setup the breadcrumb interrupt, by
a process of elimination we know the request must have been completed
and we can skip its enablement!

<4> [1518.350005] Unclaimed write to register 0x220a8
<4> [1518.350323] WARNING: CPU: 2 PID: 3685 at drivers/gpu/drm/i915/intel_uncore.c:1163 __unclaimed_reg_debug+0x40/0x50 [i915]
<4> [1518.350393] Modules linked in: vgem snd_hda_codec_hdmi x86_pkg_temp_thermal i915 coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hwdep snd_hda_core btusb cdc_ether btrtl usbnet btbcm btintel r8152 snd_pcm mii bluetooth ecdh_generic ecc i2c_hid pinctrl_sunrisepoint pinctrl_intel intel_lpss_pci prime_numbers [last unloaded: vgem]
<4> [1518.350646] CPU: 2 PID: 3685 Comm: gem_exec_parse_ Tainted: G     U            5.4.0-rc8-CI-CI_DRM_7490+ #1
<4> [1518.350708] Hardware name: Google Caroline/Caroline, BIOS MrChromebox 08/27/2018
<4> [1518.350946] RIP: 0010:__unclaimed_reg_debug+0x40/0x50 [i915]
<4> [1518.350992] Code: 74 05 5b 5d 41 5c c3 45 84 e4 48 c7 c0 95 8d 47 a0 48 c7 c6 8b 8d 47 a0 48 0f 44 f0 89 ea 48 c7 c7 9e 8d 47 a0 e8 40 45 e3 e0 <0f> 0b 83 2d 27 4f 2a 00 01 5b 5d 41 5c c3 66 90 41 55 41 54 55 53
<4> [1518.351100] RSP: 0018:ffffc900007f39c8 EFLAGS: 00010086
<4> [1518.351140] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006
<4> [1518.351202] RDX: 0000000080000006 RSI: 0000000000000000 RDI: 00000000ffffffff
<4> [1518.351249] RBP: 00000000000220a8 R08: 0000000000000000 R09: 0000000000000000
<4> [1518.351296] R10: ffffc900007f3990 R11: ffffc900007f3868 R12: 0000000000000000
<4> [1518.351342] R13: 00000000fefeffff R14: 0000000000000092 R15: ffff888155fea000
<4> [1518.351391] FS:  00007fc255abfe40(0000) GS:ffff88817ab00000(0000) knlGS:0000000000000000
<4> [1518.351445] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4> [1518.351485] CR2: 00007fc2554882d0 CR3: 0000000168ca2005 CR4: 00000000003606e0
<4> [1518.351529] Call Trace:
<4> [1518.351746]  fwtable_write32+0x114/0x1d0 [i915]
<4> [1518.351795]  ? sync_file_alloc+0x80/0x80
<4> [1518.352039]  gen8_logical_ring_enable_irq+0x30/0x50 [i915]
<4> [1518.352295]  irq_enable.part.10+0x23/0x40 [i915]
<4> [1518.352523]  i915_request_enable_breadcrumb+0xb5/0x330 [i915]
<4> [1518.352575]  ? sync_file_alloc+0x80/0x80
<4> [1518.352612]  __dma_fence_enable_signaling+0x60/0x160
<4> [1518.352653]  ? sync_file_alloc+0x80/0x80
<4> [1518.352685]  dma_fence_add_callback+0x44/0xd0
<4> [1518.352726]  sync_file_poll+0x95/0xc0
<4> [1518.352767]  do_sys_poll+0x24d/0x570
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191205215842.862750-1-chris@chris-wilson.co.uk

The function intel_engine_breadcrumbs_irq() is always invoked from an interrupt
handler and for that reason it invokes (as an optimisation) only spin_lock()
for locking assuming that the interrupts are already disabled. The
function intel_engine_signal_breadcrumbs() is provided to disable
interrupts while the former function is invoked so that assumption is
also true for callers from preemptible context.

On PREEMPT_RT local_irq_disable() really disables interrupts and this
forbids to invoke spin_lock() which becomes a sleeping spinlock.

This is also problematic with `threadirqs' in conjunction with
irq_work. With force threading the interrupt handler, the handler is
invoked with disabled BH but with interrupts enabled. This is okay and
the lock itself is never acquired in IRQ context. This changes with
irq_work (signal_irq_work()) which _still_ invokes
intel_engine_breadcrumbs_irq() from IRQ context. Lockdep should see this
and complain.

Acquire the locks in intel_engine_breadcrumbs_irq() with _irqsave()
suffix and let all callers invoke intel_engine_breadcrumbs_irq()
directly instead using intel_engine_signal_breadcrumbs().
Reported-by: NClark Williams <williams@redhat.com>
Signed-off-by: NSebastian Andrzej Siewior <bigeasy@linutronix.de>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20190926105644.16703-2-bigeasy@linutronix.de

The lockdep_assert_irqs_disabled() check is needless. The previous
lockdep_assert_held() check ensures that the lock is acquired and while
the lock is acquired lockdep also prints a warning if the interrupts are
not disabled if they have to be.
These IRQ-off asserts trigger on PREEMPT_RT because the locks become
sleeping locks and do not really disable interrupts.

Remove lockdep_assert_irqs_disabled().
Reported-by: NClark Williams <williams@redhat.com>
Signed-off-by: NSebastian Andrzej Siewior <bigeasy@linutronix.de>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20190926105644.16703-3-bigeasy@linutronix.de

The timestamp and the cb_list are mutually exclusive, the cb_list can
only be added to prior to being signaled (and once signaled we drain),
while the timestamp is only valid upon being signaled. Both the
timestamp and the cb_list are only valid while the fence is alive, and
as soon as no references are held can be replaced by the rcu_head.

By reusing the union for the timestamp, we squeeze the base dma_fence
struct to 64 bytes on x86-64.

v2: Sort the union chronologically
Suggested-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Christian König <christian.koenig@amd.com>
Acked-by: Christian König <christian.koenig@amd.com>.
Link: https://patchwork.freedesktop.org/patch/msgid/20190817153022.5749-1-chris@chris-wilson.co.uk

Stop assuming we only get called with irqs-on for disarming the
breadcrumbs, and do a full save/restore spin_lock_irq.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190813132916.20382-2-chris@chris-wilson.co.uk

The last user has been removed, so drop the functions.
Signed-off-by: NDaniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20190812233152.2172-2-daniele.ceraolospurio@intel.com

i915_irq.c is large. It serves as the central dispatch and handler for
all of our device interrupts. Lets break it up by pulling out the GT
interrupt handlers.

Based on a patch by Chris Wilson.
Signed-off-by: NAndi Shyti <andi.shyti@intel.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20190811210633.18417-1-chris@chris-wilson.co.uk

Disentangle i915_drv.h from intel_drv.h, which gets included via
i915_trace.h. This necessitates including i915_trace.h wherever it's
needed.
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NJani Nikula <jani.nikula@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/ed82bf259d3b725a1a1a3c3e9d6fb5c08bc4d489.1565085691.git.jani.nikula@intel.com

Currently there is an underlying assumption that i915_request_unsubmit()
is synchronous wrt the GPU -- that is the request is no longer in flight
as we remove it. In the near future that may change, and this may upset
our signaling as we can process an interrupt for that request while it
is no longer in flight.

CPU0					CPU1
intel_engine_breadcrumbs_irq
(queue request completion)
					i915_request_cancel_signaling
...					...
					i915_request_enable_signaling
dma_fence_signal

Hence in the time it took us to drop the lock to signal the request, a
preemption event may have occurred and re-queued the request. In the
process, that request would have seen I915_FENCE_FLAG_SIGNAL clear and
so reused the rq->signal_link that was in use on CPU0, leading to bad
pointer chasing in intel_engine_breadcrumbs_irq.

A related issue was that if someone started listening for a signal on a
completed but no longer in-flight request, we missed the opportunity to
immediately signal that request.

Furthermore, as intel_contexts may be immediately released during
request retirement, in order to be entirely sure that
intel_engine_breadcrumbs_irq may no longer dereference the intel_context
(ce->signals and ce->signal_link), we must wait for irq spinlock.

In order to prevent the race, we use a bit in the fence.flags to signal
the transfer onto the signal list inside intel_engine_breadcrumbs_irq.
For simplicity, we use the DMA_FENCE_FLAG_SIGNALED_BIT as it then
quickly signals to any outside observer that the fence is indeed signaled.

v2: Sketch out potential dma-fence API for manual signaling
v3: And the test_and_set_bit()

Fixes: 52c0fdb2 ("drm/i915: Replace global breadcrumbs with per-context interrupt tracking")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190508112452.18942-1-chris@chris-wilson.co.uk

Inside the signal handler, we expect the requests to be ordered by their
breadcrumb such that no later request may be complete if we find an
earlier incomplete. Add an assert to check that the next breadcrumb
should not be logically before the current.

v2: Move the overhanging line into its own function and reuse it after
doing the insertion.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190503152214.26517-1-chris@chris-wilson.co.uk

Start partitioning off the code that talks to the hardware (GT) from the
uapi layers and move the device facing code under gt/

One casualty is s/intel_ringbuffer.h/intel_engine.h/ with the plan to
subdivide that header and body further (and split out the submission
code from the ringbuffer and logical context handling). This patch aims
to be simple motion so git can fixup inflight patches with little mess.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Acked-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Acked-by: NJani Nikula <jani.nikula@intel.com>
Acked-by: NRodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190424174839.7141-1-chris@chris-wilson.co.uk

Since removal of the "missed interrupt detection" nobody used the result
of whether or not we signaled anybody during that invocation, so now
remove the return value.

References: 789659f4 ("drm/i915: Drop fake breadcrumb irq")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190416085218.431-1-chris@chris-wilson.co.uk

We may race the interrupt signaling with retirement, in which case the
order in which we acquire the reference inside the interrupt is vital to
provide the correct barrier against the request being freed in
retirement, i.e. we need to acquire our reference before marking the
breadcrumb as cancelled (as soon as the breadcrumb is cancelled
retirement may drop its reference to the request without serialisation
with the interrupt handler).

<3>[  683.372226] BUG i915_request (Tainted: G     U           ): Object already free
<3>[  683.372269] -----------------------------------------------------------------------------

<4>[  683.372323] Disabling lock debugging due to kernel taint
<3>[  683.372393] INFO: Allocated in i915_request_alloc+0x169/0x810 [i915] age=0 cpu=2 pid=1420
<3>[  683.372412] 	kmem_cache_alloc+0x21c/0x280
<3>[  683.372478] 	i915_request_alloc+0x169/0x810 [i915]
<3>[  683.372540] 	i915_gem_do_execbuffer+0x84e/0x1ae0 [i915]
<3>[  683.372603] 	i915_gem_execbuffer2_ioctl+0x11b/0x420 [i915]
<3>[  683.372617] 	drm_ioctl_kernel+0x83/0xf0
<3>[  683.372626] 	drm_ioctl+0x2f3/0x3b0
<3>[  683.372636] 	do_vfs_ioctl+0xa0/0x6e0
<3>[  683.372645] 	ksys_ioctl+0x35/0x60
<3>[  683.372654] 	__x64_sys_ioctl+0x11/0x20
<3>[  683.372664] 	do_syscall_64+0x55/0x190
<3>[  683.372675] 	entry_SYSCALL_64_after_hwframe+0x49/0xbe
<3>[  683.372740] INFO: Freed in i915_request_retire_upto+0xfb/0x2e0 [i915] age=0 cpu=0 pid=1419
<3>[  683.372807] 	i915_request_retire_upto+0xfb/0x2e0 [i915]
<3>[  683.372870] 	i915_request_add+0x3bd/0x9d0 [i915]
<3>[  683.372931] 	i915_gem_do_execbuffer+0x141c/0x1ae0 [i915]
<3>[  683.372991] 	i915_gem_execbuffer2_ioctl+0x11b/0x420 [i915]
<3>[  683.373001] 	drm_ioctl_kernel+0x83/0xf0
<3>[  683.373008] 	drm_ioctl+0x2f3/0x3b0
<3>[  683.373015] 	do_vfs_ioctl+0xa0/0x6e0
<3>[  683.373023] 	ksys_ioctl+0x35/0x60
<3>[  683.373030] 	__x64_sys_ioctl+0x11/0x20
<3>[  683.373037] 	do_syscall_64+0x55/0x190
<3>[  683.373045] 	entry_SYSCALL_64_after_hwframe+0x49/0xbe
<3>[  683.373054] INFO: Slab 0x0000000079bcdd71 objects=30 used=2 fp=0x000000006d77b8af flags=0x8000000000010201
<3>[  683.373069] INFO: Object 0x000000006d77b8af @offset=24000 fp=0x000000007b061eab

<3>[  683.373083] Redzone 00000000ee47ef28: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb  ................
<3>[  683.373097] Redzone 000000000cb91471: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb  ................
<3>[  683.373111] Redzone 00000000cf2b86ee: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb  ................
<3>[  683.373125] Redzone 00000000f1f5a2cd: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb  ................
<3>[  683.373139] Object 000000006d77b8af: 00 00 00 00 5a 5a 5a 5a 00 3c 49 c0 ff ff ff ff  ....ZZZZ.<I.....
<3>[  683.373153] Object 000000006f9b6204: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373167] Object 0000000091410ffb: e0 dd 6b fa 87 9f ff ff e0 dd 6b fa 87 9f ff ff  ..k.......k.....
<3>[  683.373181] Object 000000004cdf799d: 20 de 6b fa 87 9f ff ff 3d 00 00 00 00 00 00 00   .k.....=.......
<3>[  683.373195] Object 00000000545afebc: aa b3 00 00 00 00 00 00 0f 00 00 00 00 00 00 00  ................
<3>[  683.373209] Object 00000000e4a394a8: 25 bd bd 1b 9f 00 00 00 00 00 00 00 5a 5a 5a 5a  %...........ZZZZ
<3>[  683.373223] Object 0000000029a7878a: 00 00 00 00 ad 4e ad de ff ff ff ff 5a 5a 5a 5a  .....N......ZZZZ
<3>[  683.373237] Object 00000000d37797b3: ff ff ff ff ff ff ff ff e8 6e 57 c0 ff ff ff ff  .........nW.....
<3>[  683.373251] Object 00000000d50414f6: 00 b3 c8 8e ff ff ff ff 80 b0 c8 8e ff ff ff ff  ................
<3>[  683.373265] Object 00000000c28e8847: 41 01 4b c0 ff ff ff ff 00 00 88 8e 88 9f ff ff  A.K.............
<3>[  683.373279] Object 00000000c74212ab: 38 c1 6d 8a 88 9f ff ff 58 21 74 8a 88 9f ff ff  8.m.....X!t.....
<3>[  683.373293] Object 000000000d8012cf: c0 c1 6d 8a 88 9f ff ff 58 79 dd d9 87 9f ff ff  ..m.....Xy......
<3>[  683.373306] Object 00000000c9900b91: 98 d0 4e 8a 88 9f ff ff 58 3c e8 9b 88 9f ff ff  ..N.....X<......
<3>[  683.373320] Object 0000000044bb8c3d: 58 3c e8 9b 88 9f ff ff 64 f5 04 00 00 00 00 00  X<......d.......
<3>[  683.373334] Object 00000000180c4cca: 00 00 00 00 ad 4e ad de ff ff ff ff 5a 5a 5a 5a  .....N......ZZZZ
<3>[  683.373348] Object 00000000c9044498: ff ff ff ff ff ff ff ff e0 6e 57 c0 ff ff ff ff  .........nW.....
<3>[  683.373362] Object 0000000072d0dfb3: 00 00 00 00 00 00 00 00 c0 b1 c8 8e ff ff ff ff  ................
<3>[  683.373376] Object 0000000081f198b9: 55 01 4b c0 ff ff ff ff d8 de 6b fa 87 9f ff ff  U.K.......k.....
<3>[  683.373390] Object 000000006a375a13: d8 de 6b fa 87 9f ff ff cc 05 39 c0 ff ff ff ff  ..k.......9.....
<3>[  683.373404] Object 00000000b8392dd1: ff ff ff ff 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ....ZZZZZZZZZZZZ
<3>[  683.373418] Object 00000000e5c1bbcb: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373432] Object 00000000199feccd: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373446] Object 0000000020f5e08b: 20 df 6b fa 87 9f ff ff 20 df 6b fa 87 9f ff ff   .k..... .k.....
<3>[  683.373460] Object 0000000090591b0f: 30 df 6b fa 87 9f ff ff 30 df 6b fa 87 9f ff ff  0.k.....0.k.....
<3>[  683.373473] Object 00000000232f7cd0: 40 df 6b fa 87 9f ff ff 40 df 6b fa 87 9f ff ff  @.k.....@.k.....
<3>[  683.373487] Object 0000000060458027: 50 df 6b fa 87 9f ff ff 50 df 6b fa 87 9f ff ff  P.k.....P.k.....
<3>[  683.373501] Object 00000000e3c82ce2: 06 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
<3>[  683.373515] Object 00000000ec804eb8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373529] Object 00000000ce7ccc08: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373543] Object 000000002dbc575c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373557] Object 00000000b86d3417: 5a 5a 5a 5a 5a 5a 5a 5a 00 de 6b fa 87 9f ff ff  ZZZZZZZZ..k.....
<3>[  683.373571] Object 00000000d1e82276: b8 61 dd d9 87 9f ff ff a0 06 00 00 d0 06 00 00  .a..............
<3>[  683.373585] Object 00000000cc53f969: e8 06 00 00 20 07 00 00 28 07 00 00 00 00 00 00  .... ...(.......
<3>[  683.373599] Object 00000000ea2426d2: 40 0c 8c 7b 88 9f ff ff 00 00 00 00 00 00 00 00  @..{............
<3>[  683.373613] Object 00000000b860c1c3: 68 0d 8c 7b 88 9f ff ff 68 25 8c 7b 88 9f ff ff  h..{....h%.{....
<3>[  683.373627] Object 0000000016455ea0: 96 d5 05 00 01 00 00 00 00 5a 5a 5a 5a 5a 5a 5a  .........ZZZZZZZ
<3>[  683.373640] Object 00000000e66ede82: 00 e0 6b fa 87 9f ff ff 00 e0 6b fa 87 9f ff ff  ..k.......k.....
<3>[  683.373654] Object 0000000080964939: 10 e0 6b fa 87 9f ff ff 10 e0 6b fa 87 9f ff ff  ..k.......k.....
<3>[  683.373668] Object 00000000e7ffc5dd: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
<3>[  683.373682] Object 000000000ce9d6ca: 00 02 00 00 00 00 ad de 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
<3>[  683.373696] Object 00000000386659d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373710] Redzone 0000000075d2069d: bb bb bb bb bb bb bb bb                          ........
<3>[  683.373723] Padding 0000000054e14c6b: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373737] Padding 00000000425e5b34: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373751] Padding 00000000ad3d4db9: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<4>[  683.373767] CPU: 1 PID: 151 Comm: kworker/1:2 Tainted: G    BU            5.0.0-rc8-g39139489403b-drmtip_236+ #1
<4>[  683.373769] Hardware name: Intel Corporation Ice Lake Client Platform/IceLake Y LPDDR4x T4 RVP TLC, BIOS ICLSFWR1.R00.3087.A00.1902250334 02/25/2019
<4>[  683.373773] Workqueue: events delayed_fput
<4>[  683.373775] Call Trace:
<4>[  683.373777]  <IRQ>
<4>[  683.373781]  dump_stack+0x67/0x9b
<4>[  683.373783]  free_debug_processing+0x344/0x370
<4>[  683.373832]  ? intel_engine_breadcrumbs_irq+0x2e4/0x380 [i915]
<4>[  683.373836]  __slab_free+0x337/0x4f0
<4>[  683.373840]  ? _raw_spin_unlock_irqrestore+0x39/0x60
<4>[  683.373844]  ? debug_check_no_obj_freed+0x132/0x210
<4>[  683.373889]  ? intel_engine_breadcrumbs_irq+0x2e4/0x380 [i915]
<4>[  683.373892]  ? kmem_cache_free+0x275/0x2e0
<4>[  683.373894]  kmem_cache_free+0x275/0x2e0
<4>[  683.373939]  intel_engine_breadcrumbs_irq+0x2e4/0x380 [i915]
<4>[  683.373984]  gen8_cs_irq_handler+0x4e/0xa0 [i915]
<4>[  683.374026]  gen11_irq_handler+0x24b/0x330 [i915]
<4>[  683.374032]  __handle_irq_event_percpu+0x41/0x2d0
<4>[  683.374034]  ? handle_irq_event+0x27/0x50
<4>[  683.374038]  handle_irq_event_percpu+0x2b/0x70
<4>[  683.374040]  handle_irq_event+0x2f/0x50
<4>[  683.374044]  handle_edge_irq+0xe7/0x190
<4>[  683.374048]  handle_irq+0x67/0x160
<4>[  683.374051]  do_IRQ+0x5e/0x130
<4>[  683.374054]  common_interrupt+0xf/0xf

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=109827
Fixes: 52c0fdb2 ("drm/i915: Replace global breadcrumbs with per-context interrupt tracking")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190304114113.371-1-chris@chris-wilson.co.uk
(cherry picked from commit e781a7a3)
Signed-off-by: NRodrigo Vivi <rodrigo.vivi@intel.com>

We may race the interrupt signaling with retirement, in which case the
order in which we acquire the reference inside the interrupt is vital to
provide the correct barrier against the request being freed in
retirement, i.e. we need to acquire our reference before marking the
breadcrumb as cancelled (as soon as the breadcrumb is cancelled
retirement may drop its reference to the request without serialisation
with the interrupt handler).

<3>[  683.372226] BUG i915_request (Tainted: G     U           ): Object already free
<3>[  683.372269] -----------------------------------------------------------------------------

<4>[  683.372323] Disabling lock debugging due to kernel taint
<3>[  683.372393] INFO: Allocated in i915_request_alloc+0x169/0x810 [i915] age=0 cpu=2 pid=1420
<3>[  683.372412] 	kmem_cache_alloc+0x21c/0x280
<3>[  683.372478] 	i915_request_alloc+0x169/0x810 [i915]
<3>[  683.372540] 	i915_gem_do_execbuffer+0x84e/0x1ae0 [i915]
<3>[  683.372603] 	i915_gem_execbuffer2_ioctl+0x11b/0x420 [i915]
<3>[  683.372617] 	drm_ioctl_kernel+0x83/0xf0
<3>[  683.372626] 	drm_ioctl+0x2f3/0x3b0
<3>[  683.372636] 	do_vfs_ioctl+0xa0/0x6e0
<3>[  683.372645] 	ksys_ioctl+0x35/0x60
<3>[  683.372654] 	__x64_sys_ioctl+0x11/0x20
<3>[  683.372664] 	do_syscall_64+0x55/0x190
<3>[  683.372675] 	entry_SYSCALL_64_after_hwframe+0x49/0xbe
<3>[  683.372740] INFO: Freed in i915_request_retire_upto+0xfb/0x2e0 [i915] age=0 cpu=0 pid=1419
<3>[  683.372807] 	i915_request_retire_upto+0xfb/0x2e0 [i915]
<3>[  683.372870] 	i915_request_add+0x3bd/0x9d0 [i915]
<3>[  683.372931] 	i915_gem_do_execbuffer+0x141c/0x1ae0 [i915]
<3>[  683.372991] 	i915_gem_execbuffer2_ioctl+0x11b/0x420 [i915]
<3>[  683.373001] 	drm_ioctl_kernel+0x83/0xf0
<3>[  683.373008] 	drm_ioctl+0x2f3/0x3b0
<3>[  683.373015] 	do_vfs_ioctl+0xa0/0x6e0
<3>[  683.373023] 	ksys_ioctl+0x35/0x60
<3>[  683.373030] 	__x64_sys_ioctl+0x11/0x20
<3>[  683.373037] 	do_syscall_64+0x55/0x190
<3>[  683.373045] 	entry_SYSCALL_64_after_hwframe+0x49/0xbe
<3>[  683.373054] INFO: Slab 0x0000000079bcdd71 objects=30 used=2 fp=0x000000006d77b8af flags=0x8000000000010201
<3>[  683.373069] INFO: Object 0x000000006d77b8af @offset=24000 fp=0x000000007b061eab

<3>[  683.373083] Redzone 00000000ee47ef28: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb  ................
<3>[  683.373097] Redzone 000000000cb91471: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb  ................
<3>[  683.373111] Redzone 00000000cf2b86ee: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb  ................
<3>[  683.373125] Redzone 00000000f1f5a2cd: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb  ................
<3>[  683.373139] Object 000000006d77b8af: 00 00 00 00 5a 5a 5a 5a 00 3c 49 c0 ff ff ff ff  ....ZZZZ.<I.....
<3>[  683.373153] Object 000000006f9b6204: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373167] Object 0000000091410ffb: e0 dd 6b fa 87 9f ff ff e0 dd 6b fa 87 9f ff ff  ..k.......k.....
<3>[  683.373181] Object 000000004cdf799d: 20 de 6b fa 87 9f ff ff 3d 00 00 00 00 00 00 00   .k.....=.......
<3>[  683.373195] Object 00000000545afebc: aa b3 00 00 00 00 00 00 0f 00 00 00 00 00 00 00  ................
<3>[  683.373209] Object 00000000e4a394a8: 25 bd bd 1b 9f 00 00 00 00 00 00 00 5a 5a 5a 5a  %...........ZZZZ
<3>[  683.373223] Object 0000000029a7878a: 00 00 00 00 ad 4e ad de ff ff ff ff 5a 5a 5a 5a  .....N......ZZZZ
<3>[  683.373237] Object 00000000d37797b3: ff ff ff ff ff ff ff ff e8 6e 57 c0 ff ff ff ff  .........nW.....
<3>[  683.373251] Object 00000000d50414f6: 00 b3 c8 8e ff ff ff ff 80 b0 c8 8e ff ff ff ff  ................
<3>[  683.373265] Object 00000000c28e8847: 41 01 4b c0 ff ff ff ff 00 00 88 8e 88 9f ff ff  A.K.............
<3>[  683.373279] Object 00000000c74212ab: 38 c1 6d 8a 88 9f ff ff 58 21 74 8a 88 9f ff ff  8.m.....X!t.....
<3>[  683.373293] Object 000000000d8012cf: c0 c1 6d 8a 88 9f ff ff 58 79 dd d9 87 9f ff ff  ..m.....Xy......
<3>[  683.373306] Object 00000000c9900b91: 98 d0 4e 8a 88 9f ff ff 58 3c e8 9b 88 9f ff ff  ..N.....X<......
<3>[  683.373320] Object 0000000044bb8c3d: 58 3c e8 9b 88 9f ff ff 64 f5 04 00 00 00 00 00  X<......d.......
<3>[  683.373334] Object 00000000180c4cca: 00 00 00 00 ad 4e ad de ff ff ff ff 5a 5a 5a 5a  .....N......ZZZZ
<3>[  683.373348] Object 00000000c9044498: ff ff ff ff ff ff ff ff e0 6e 57 c0 ff ff ff ff  .........nW.....
<3>[  683.373362] Object 0000000072d0dfb3: 00 00 00 00 00 00 00 00 c0 b1 c8 8e ff ff ff ff  ................
<3>[  683.373376] Object 0000000081f198b9: 55 01 4b c0 ff ff ff ff d8 de 6b fa 87 9f ff ff  U.K.......k.....
<3>[  683.373390] Object 000000006a375a13: d8 de 6b fa 87 9f ff ff cc 05 39 c0 ff ff ff ff  ..k.......9.....
<3>[  683.373404] Object 00000000b8392dd1: ff ff ff ff 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ....ZZZZZZZZZZZZ
<3>[  683.373418] Object 00000000e5c1bbcb: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373432] Object 00000000199feccd: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373446] Object 0000000020f5e08b: 20 df 6b fa 87 9f ff ff 20 df 6b fa 87 9f ff ff   .k..... .k.....
<3>[  683.373460] Object 0000000090591b0f: 30 df 6b fa 87 9f ff ff 30 df 6b fa 87 9f ff ff  0.k.....0.k.....
<3>[  683.373473] Object 00000000232f7cd0: 40 df 6b fa 87 9f ff ff 40 df 6b fa 87 9f ff ff  @.k.....@.k.....
<3>[  683.373487] Object 0000000060458027: 50 df 6b fa 87 9f ff ff 50 df 6b fa 87 9f ff ff  P.k.....P.k.....
<3>[  683.373501] Object 00000000e3c82ce2: 06 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
<3>[  683.373515] Object 00000000ec804eb8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373529] Object 00000000ce7ccc08: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373543] Object 000000002dbc575c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373557] Object 00000000b86d3417: 5a 5a 5a 5a 5a 5a 5a 5a 00 de 6b fa 87 9f ff ff  ZZZZZZZZ..k.....
<3>[  683.373571] Object 00000000d1e82276: b8 61 dd d9 87 9f ff ff a0 06 00 00 d0 06 00 00  .a..............
<3>[  683.373585] Object 00000000cc53f969: e8 06 00 00 20 07 00 00 28 07 00 00 00 00 00 00  .... ...(.......
<3>[  683.373599] Object 00000000ea2426d2: 40 0c 8c 7b 88 9f ff ff 00 00 00 00 00 00 00 00  @..{............
<3>[  683.373613] Object 00000000b860c1c3: 68 0d 8c 7b 88 9f ff ff 68 25 8c 7b 88 9f ff ff  h..{....h%.{....
<3>[  683.373627] Object 0000000016455ea0: 96 d5 05 00 01 00 00 00 00 5a 5a 5a 5a 5a 5a 5a  .........ZZZZZZZ
<3>[  683.373640] Object 00000000e66ede82: 00 e0 6b fa 87 9f ff ff 00 e0 6b fa 87 9f ff ff  ..k.......k.....
<3>[  683.373654] Object 0000000080964939: 10 e0 6b fa 87 9f ff ff 10 e0 6b fa 87 9f ff ff  ..k.......k.....
<3>[  683.373668] Object 00000000e7ffc5dd: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
<3>[  683.373682] Object 000000000ce9d6ca: 00 02 00 00 00 00 ad de 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
<3>[  683.373696] Object 00000000386659d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373710] Redzone 0000000075d2069d: bb bb bb bb bb bb bb bb                          ........
<3>[  683.373723] Padding 0000000054e14c6b: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373737] Padding 00000000425e5b34: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<3>[  683.373751] Padding 00000000ad3d4db9: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
<4>[  683.373767] CPU: 1 PID: 151 Comm: kworker/1:2 Tainted: G    BU            5.0.0-rc8-g39139489403b-drmtip_236+ #1
<4>[  683.373769] Hardware name: Intel Corporation Ice Lake Client Platform/IceLake Y LPDDR4x T4 RVP TLC, BIOS ICLSFWR1.R00.3087.A00.1902250334 02/25/2019
<4>[  683.373773] Workqueue: events delayed_fput
<4>[  683.373775] Call Trace:
<4>[  683.373777]  <IRQ>
<4>[  683.373781]  dump_stack+0x67/0x9b
<4>[  683.373783]  free_debug_processing+0x344/0x370
<4>[  683.373832]  ? intel_engine_breadcrumbs_irq+0x2e4/0x380 [i915]
<4>[  683.373836]  __slab_free+0x337/0x4f0
<4>[  683.373840]  ? _raw_spin_unlock_irqrestore+0x39/0x60
<4>[  683.373844]  ? debug_check_no_obj_freed+0x132/0x210
<4>[  683.373889]  ? intel_engine_breadcrumbs_irq+0x2e4/0x380 [i915]
<4>[  683.373892]  ? kmem_cache_free+0x275/0x2e0
<4>[  683.373894]  kmem_cache_free+0x275/0x2e0
<4>[  683.373939]  intel_engine_breadcrumbs_irq+0x2e4/0x380 [i915]
<4>[  683.373984]  gen8_cs_irq_handler+0x4e/0xa0 [i915]
<4>[  683.374026]  gen11_irq_handler+0x24b/0x330 [i915]
<4>[  683.374032]  __handle_irq_event_percpu+0x41/0x2d0
<4>[  683.374034]  ? handle_irq_event+0x27/0x50
<4>[  683.374038]  handle_irq_event_percpu+0x2b/0x70
<4>[  683.374040]  handle_irq_event+0x2f/0x50
<4>[  683.374044]  handle_edge_irq+0xe7/0x190
<4>[  683.374048]  handle_irq+0x67/0x160
<4>[  683.374051]  do_IRQ+0x5e/0x130
<4>[  683.374054]  common_interrupt+0xf/0xf

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=109827
Fixes: 52c0fdb2 ("drm/i915: Replace global breadcrumbs with per-context interrupt tracking")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190304114113.371-1-chris@chris-wilson.co.uk

Missed breadcrumb detection is defunct due to the tight coupling with
dma_fence signaling and the myriad ways we may signal fences from
everywhere but from an interrupt, i.e. we frequently signal a fence
before we even see its interrupt. This means that even if we miss an
interrupt for a fence, it still is signaled before our breadcrumb
hangcheck fires, so simplify the breadcrumb hangchecking by moving it
into the GPU hangcheck and forgo fake interrupts.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-3-chris@chris-wilson.co.uk

A few years ago, see commit 688e6c72 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.

To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.

Before commit 688e6c72, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c72 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)

The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).

Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.

v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.

References: 688e6c72 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk

A repeated pattern is to test the signaled bit of our
request->fence.flags. Make this an inline to shorten a few lines and
remove unnecessary line continuations.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190121222117.23305-20-chris@chris-wilson.co.uk

Since commit  d4ccceb0 ("drm/i915/icl: Ringbuffer interrupt handling")
we have required a mechanism to avoid touching the interrupt hardware
for breadcrumbs, superseding our mock interface for selftests.

The residual problem (ideas welcome) is in probing the mock ring
registers for ring_is_idle. Hmm, maybe we should just install
mock handlers for i915->uncore.mmio__write and friends? Only problem
being is that we would to truly mock some expected reads. :(

References: d4ccceb0 ("drm/i915/icl: Ringbuffer interrupt handling")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190118112225.13780-1-chris@chris-wilson.co.uk

The motivation for introducing the check that we only enable breadcrumb
irqs if the device's irq was installed was once upon a time we waited
during suspend after disabling interrupts (which was quite slow until
the bug was discovered). Since then we have the notion of pinning the
breadcrumb irq, broadening it from the sole purpose of user interrupt
notification and waiting, and more importantly decoupling it from a very
defined time period during which enabling the irq was expected. So stop
insisting the irq is installed before we setup our IMR masks, if the IER
isn't yet enabled, nothing will happen and we will timeout instead,
revealing the lack of irq in the hang debug messages.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190117233126.30165-1-chris@chris-wilson.co.uk

Now that we have eliminated the CPU-side irq_seqno_barrier by moving the
delays on the GPU before emitting the MI_USER_INTERRUPT, we can remove
the engine->irq_seqno_barrier infrastructure. Though intentionally
slowing down the GPU is nasty, so is the code we can now remove!
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20181228171641.16531-6-chris@chris-wilson.co.uk

Change the on-cpu check to on-runqueue to catch if the waiter has been
woken (and reset its current_state back to TASK_UNINTERRUPTIBLE to
perform the seqno check) but is sleeping due to being preempted off the
cpu.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20181203113701.12106-1-chris@chris-wilson.co.uk

We have a few instances of checking seqno-1 to see if the HW has started
the request. Pull those together under a helper.

v2: Pull the !seqno assertion higher, as given seqno==1 we may indeed
check to see if we have started using seqno==0.
Suggested-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180806112605.20725-1-chris@chris-wilson.co.uk

Rather than have multiple locked instructions inside the notify_ring()
irq handler, move them inside the spinlock and reduce their intrinsic
locking.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180627201304.15817-3-chris@chris-wilson.co.uk

In order to support engine reset from irq (timer) context, we need to be
able to re-initialise the breadcrumbs. So we need to promote the plain
spin_lock_irq to a safe spin_lock_irqsave.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180518090212.5349-3-chris@chris-wilson.co.uk

The majority of the engine state dumping is too voluminous to be useful
outside of a controlled setup, though a few do accompany severe errors.
Keep the debug dumps next to the errors, but hide the others behind a CI
compile flag. This becomes more useful when adding more dumps to latency
sensitive paths.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180426103219.22181-1-chris@chris-wilson.co.uk

Instead of synchronously cancelling the timer and re-enabling it inside
the reset callbacks, keep the timer enabled and let it die on its next
wakeup if no longer required. This allows
intel_engine_reset_breadcrumbs() to be used from an atomic
(timer/softirq) context such as required for resetting an engine.

It also allows us to react better to the user poking around debugfs for
testing missed irqs.

v2: Tighten the order of del_timer_sync as the fake_irq timer
may trigger the hangcheck timer, and so we should cancel it first and
then cancel the hangcheck (Mika)
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180424142945.6787-1-chris@chris-wilson.co.uk

There is some redundancy between dma_fence->ops->enable_signaling (via
i915_fence_enable_signaling) and our backend,
intel_engine_enable_signaling() in that both levels recheck the fence
status multiple times. If we convert intel_engine_enable_signaling() to
return the information desired by dma_fence->ops->enable_signaling, we
can reduce i915_fence_enable_signaling to a simple stub and avoid
trying to reinterpret the same information.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Reviewed-by: NTvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180308140732.25090-1-chris@chris-wilson.co.uk

Previously, we would spin waiting for all waiters to wake up and notice
their request had completed before we would reset the seqno upon
wraparound.  However, we can mark their waits as complete and wake them
up directly using the existing machinery for handling the flushing of
missed wakeups when idling.
Suggested-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180306130143.13312-2-chris@chris-wilson.co.uk

Since commit fd10e2ce ("drm/i915/breadcrumbs: Ignore unsubmitted
signalers"), we cancel the signaler when retiring the request and so
upon wraparound, where we wait for all requests to be retired, we no
longer need to spin waiting for the signaling thread to release its
references to the in-flight requests, and so we can assert that the
signaler is idle.

References: fd10e2ce ("drm/i915/breadcrumbs: Ignore unsubmitted signalers")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180306130143.13312-1-chris@chris-wilson.co.uk

When parking the engines and their breadcrumbs, if we have waiters left
then they missed their wakeup. Verify that each waiter's seqno did
complete.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180222092545.17216-2-chris@chris-wilson.co.uk