Factor out these two interfaces so we can kill some duplicated code in
scheduler.c.

v2:
  - rename to intel_vgpu_{get,put}_ppgtt_mm
  - refine handle_g2v_notification
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Reviewed-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

If we manage an object with a reference count, then its life cycle
must flow the reference count operations. Meanwhile, change the
operation functions to generic name *put* and *get*.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

This is a big one and the GVT shadow graphic memory management code is
heavily refined. The new code is more straightforward with less code.

The struct intel_vgpu_mm is restructured to be clearly defined, use
accurate names and some of the original fields are removed which are
really redundant.

Now we only manage ppgtt mm object with mm->ppgtt_mm.lru_list. No need
to mix ppgtt and ggtt together, since one vGPU only has one ggtt object.

v4: Don't invoke ppgtt_free_all_shadow_page before intel_vgpu_destroy_all_ppgtt_mm.
v3: Add GVT_RING_CTX_NR_PDPS to avoid confusing about the PDPs.
v2: Split some changes into small standalone patches.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

We want to de-emphasize the link between the request (dependency,
execution and fence tracking) from GEM and so rename the struct from
drm_i915_gem_request to i915_request. That is we may implement the GEM
user interface on top of requests, but they are an abstraction for
tracking execution rather than an implementation detail of GEM. (Since
they are not tied to HW, we keep the i915 prefix as opposed to intel.)

In short, the spatch:
@@

@@
- struct drm_i915_gem_request
+ struct i915_request

A corollary to contracting the type name, we also harmonise on using
'rq' shorthand for local variables where space if of the essence and
repetition makes 'request' unwieldy. For globals and struct members,
'request' is still much preferred for its clarity.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Michal Wajdeczko <michal.wajdeczko@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180221095636.6649-1-chris@chris-wilson.co.ukReviewed-by: NMika Kuoppala <mika.kuoppala@linux.intel.com>
Reviewed-by: NMichał Winiarski <michal.winiarski@intel.com>
Acked-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>

Only reset vgpu execlist state of the exact engine which gets reset
request from VM. After read context status from HWSP enabled, KMD will use
the saved CSB read pointer but not always read from MMIO. When one engine
reset happen, only the read pointer of this engine will be reset, in GVT-g
host side also need to align with this policy, otherwise VM may get wrong
CSB status after one engine reset compeleted.

v2: Split refine and fix patch, code refine(Zhenyu)
v3: Move active flag of vgpu scheduler into sched_data(Zhenyu)

Cc: Fred Gao <fred.gao@intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NWeinan Li <weinan.z.li@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NRodrigo Vivi <rodrigo.vivi@intel.com>

Using per engine ops will be more flexible, here refine sub-ops(init,
clean) as per engine operation align with reset operation. This change also
will be used in next fix patch for VM engine reset.

Cc: Fred Gao <fred.gao@intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NWeinan Li <weinan.z.li@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NRodrigo Vivi <rodrigo.vivi@intel.com>

This is to workaround guest driver hang regression after
preemption enable that gvt hasn't enabled handling of that
for guest workload. So in effect this disables preemption
for gvt context now.
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>
(cherry picked from commit 1603660b)

We shouldn't mark inactive for vGPU context if preempted,
which would still be re-scheduled later. So keep active state.

Fixes: d6c05113 ("drm/i915/execlists: Distinguish the incomplete context notifies")
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>
(cherry picked from commit da5f99ea)

mmio_read_from_hw() let vgpu could read hw reg, if vgpu's workload
is running on hw, things is good. Otherwise vgpu will get other
vgpu's reg val, it is unsafe.

This patch limit such hw access to active vgpu. If vgpu isn't
running on hw, the reg read of this vgpu will get the last active
val which saved at schedule_out.

v2: ring timestamp is walking continuously even if the ring is idle.
    so read hw directly. (Zhenyu)
Signed-off-by: NXiong Zhang <xiong.y.zhang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>
(cherry picked from commit 295764cd)

This is to workaround guest driver hang regression after
preemption enable that gvt hasn't enabled handling of that
for guest workload. So in effect this disables preemption
for gvt context now.
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

We shouldn't mark inactive for vGPU context if preempted,
which would still be re-scheduled later. So keep active state.

Fixes: d6c05113 ("drm/i915/execlists: Distinguish the incomplete context notifies")
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

The current schedule policy rely on a 1ms timer to execute workload. This
can introduce maximum 1ms unnecessary latency. This is especially bad for
small media workloads.

And I don't think we need this timer for QoS, but the change is not simply
remove the code. So I made a new API intel_gvt_kick_schedule() for future
change.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Convert the macro to a function which should always be preferred.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Previously the performance is improved through the workload auditing
and shadowing ahead of vGPU scheduling, however, there is the case that
more requests are allocated in submit_context before the previous request
is added, the timeline will hold its seqno which is later.

This patch is to move the request alloc to dispatch_workload function,
where is the same place as request is added.

It will fix the issue of kernel BUG for (timeline->seqno != request->fence.seqno)
check when add_request.

Fixes: 89ea20b9 ("drm/i915/gvt: Factor out scan and shadow from workload dispatch")
Signed-off-by: NChuanxiao Dong <chuanxiao.dong@intel.com>
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>
(cherry picked from commit f2880e04)

The caller of shadow_context_status_change may disable irqs. So it is not
safe to use spin_unlock_bh in such context. Let's switch to irqsave version
for safety.

------------[ cut here ]------------
WARNING: CPU: 2 PID: 4504 at kernel/softirq.c:161 __local_bh_enable_ip+0x46/0x60
[  168.797710] Hardware name: Dell Inc. OptiPlex 7040/0Y7WYT, BIOS 1.2.8 01/26/2016
[  168.797712] task: ffff8c693d22db80 task.stack: ffffb51b482bc000
[  168.797718] RIP: 0010:__local_bh_enable_ip+0x46/0x60
[  168.797721] RSP: 0018:ffffb51b482bfa10 EFLAGS: 00010046
[  168.797724] RAX: 0000000000000046 RBX: ffff8c6900278000 RCX: 00000000ffffffff
[  168.797726] RDX: 0000000000000001 RSI: 0000000000000200 RDI: ffffffffc06a0330
[  168.797728] RBP: ffffb51b482bfa10 R08: 0000000000000000 R09: ffff8c690027cb90
[  168.797730] R10: ffffb51b482bfa40 R11: 00000004072f0001 R12: 0000000000000000
[  168.797732] R13: 0000000000000000 R14: ffff8c690027ca9c R15: 0000000000000000
[  168.797735] FS:  00007ff187c56700(0000) GS:ffff8c6959d00000(0000) knlGS:0000000000000000
[  168.797738] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  168.797740] CR2: 0000562bc0c3991f CR3: 0000000430614006 CR4: 00000000003606e0
[  168.797742] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  168.797744] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  168.797745] Call Trace:
[  168.797755]  _raw_spin_unlock_bh+0x1e/0x20
[  168.797826]  shadow_context_status_change+0x120/0x1e0 [i915]
[  168.797831]  notifier_call_chain+0x4a/0x70
[  168.797834]  atomic_notifier_call_chain+0x1a/0x20
[  168.797896]  execlists_cancel_port_requests+0x4f/0x80 [i915]
[  168.797956]  reset_common_ring+0x30/0x100 [i915]
[  168.798007]  i915_gem_reset_engine+0x114/0x330 [i915]
[  168.798060]  ? i915_gem_retire_requests+0x75/0x180 [i915]
[  168.798111]  i915_gem_reset+0x3e/0xb0 [i915]
[  168.798149]  i915_reset+0x10b/0x1c0 [i915]
[  168.798187]  i915_reset_device+0x209/0x220 [i915]
[  168.798225]  ? gen8_gt_irq_ack+0x170/0x170 [i915]
[  168.798229]  ? __queue_work+0x430/0x430
[  168.798270]  i915_handle_error+0x285/0x420 [i915]
[  168.798275]  ? mntput+0x24/0x40
[  168.798281]  ? terminate_walk+0x8e/0xf0
[  168.798328]  i915_wedged_set+0x84/0xc0 [i915]
[  168.798333]  simple_attr_write+0xab/0xc0
[  168.798337]  full_proxy_write+0x54/0x90
[  168.798343]  __vfs_write+0x37/0x170
[  168.798349]  ? common_file_perm+0x4c/0x100
[  168.798355]  ? apparmor_file_permission+0x1a/0x20
[  168.798361]  ? security_file_permission+0x3b/0xc0
[  168.798365]  vfs_write+0xb8/0x1b0
[  168.798370]  SyS_write+0x55/0xc0
[  168.798376]  entry_SYSCALL_64_fastpath+0x1e/0xa9

Fixes: 0e86cc9c ("drm/i915/gvt: implement per-vm mmio switching optimization")
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

drivers/gpu/drm/i915/gvt/execlist.c:531:6: warning: symbol 'clean_execlist' was not declared. Should it be static?
drivers/gpu/drm/i915/gvt/execlist.c:545:6: warning: symbol 'reset_execlist' was not declared. Should it be static?
drivers/gpu/drm/i915/gvt/execlist.c:556:5: warning: symbol 'init_execlist' was not declared. Should it be static?
drivers/gpu/drm/i915/gvt/scheduler.c:248:6: warning: symbol 'release_shadow_wa_ctx' was not declared. Should it be static?

References: 06bb372f ("drm/i915/gvt: Introduce intel_vgpu_reset_submission")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: intel-gvt-dev@lists.freedesktop.org
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Previously the performance is improved through the workload auditing
and shadowing ahead of vGPU scheduling, however, there is the case that
more requests are allocated in submit_context before the previous request
is added, the timeline will hold its seqno which is later.

This patch is to move the request alloc to dispatch_workload function,
where is the same place as request is added.

It will fix the issue of kernel BUG for (timeline->seqno != request->fence.seqno)
check when add_request.

Fixes: 89ea20b9 ("drm/i915/gvt: Factor out scan and shadow from workload dispatch")
Signed-off-by: NChuanxiao Dong <chuanxiao.dong@intel.com>
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

mmio_read_from_hw() let vgpu could read hw reg, if vgpu's workload
is running on hw, things is good. Otherwise vgpu will get other
vgpu's reg val, it is unsafe.

This patch limit such hw access to active vgpu. If vgpu isn't
running on hw, the reg read of this vgpu will get the last active
val which saved at schedule_out.

v2: ring timestamp is walking continuously even if the ring is idle.
    so read hw directly. (Zhenyu)
Signed-off-by: NXiong Zhang <xiong.y.zhang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

As there is already an I915_GTT_PAGE_SIZE marco in i915, let GVT-g use it
as well. Also this patch re-names some GTT marcos with additional prefix.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

1) Use standard i915 GEM object sequence to access the shadow batch buffer.
2) Manage i915 vma life cycle to solve one FIXME.

v2:
- Refine code structure.
- Refine the usage of GEM APIs.
- Add the missing lock/unlock in release_shadow_batch_buffer.

Test on my SKL NuC.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Move clean_workloads() into scheduler.c since it's not specific to
execlist.

v2:

- Remove clean_workloads in intel_vgpu_select_submission_ops. (Zhenyu)
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Introduce an generic API to reset vGPU virtual submission interface.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Introduce vGPU submission ops to support easy switching submission mode
of one vGPU between different OSes.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Move common vGPU workload creation functions into scheduler.c since
they are not specific to execlist emulation.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Move common workload preparation into prepare_workload() in scheduler.c,
as they are not specific to execlist emulation.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Factor out prepare_workload() for the following re-factor.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Factor out vGPU workload creation/destroy functions since they are not
specific to execlist emulation.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

When a scan error occurs in dispatch_workload, this patch is to
check the healthy state and free all the queued workloads before
the failsafe mode is entered.
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Generally, there are 3 types of errors during command scan: a) some
commands might be unknown with EBADRQC;  b) some cmd access invalid
address with EFAULT; c) some unexpected force nonpriv cmd with EPERM.
later the healthy state can be judged through the return error.

v2:
- remove some internal i915 errors rating.  (Zhenyu)

v3:
- the healthy state is judged through the internal defined return
  error. (Zhenyu)
- force non priv cmd error can be ignored. (Kevin)

v4:
- reuse standard defined errno instead of recreate, e.g EBADRQC for
  unknown cmd, EFAULT for invalid address, EPERM for nonpriv. (Zhenyu)

v5:
- remove some irrelevant code for the patch.
- fix typo of vgpu_is_vm_unhealthy. (Zhenyu)

v6:
- move the healthy check and failsafe code into another patch. (Zhenyu)

v7:
- polish title and commit message. (Zhenyu)
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Move tlb_handle_pending into intel_vgpu_submssion since it belongs to a
part of vGPU submission stuffs
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Introduce intel_vgpu_submission to hold all members related to submission
in struct intel_vgpu before.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Move vGPU workload cache initialization/de-initialization into
intel_vgpu_{setup, clean}_submission() since they are not specific to
execlist stuffs.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

To move workload related functions into scheduler.c, an expected way is
to collect all the init/clean functions related to vGPU workload
submission into fewer functions.

Rename intel_vgpu_{init, clean}_gvt_context() for above usage in future.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Take a copy of the HW state after a reset upon module loading by
executing a context switch from a blank context to the kernel context,
thus saving the default hw state over the blank context image.
We can then use the default hw state to initialise any future context,
ensuring that each starts with the default view of hw state.

v2: Unmap our default state from the GTT after stealing it from the
context. This should stop us from accidentally overwriting it via the
GTT (and frees up some precious GTT space).

Testcase: igt/gem_ctx_isolation
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171110142634.10551-7-chris@chris-wilson.co.uk

Let the listener know that the context we just scheduled out was not
complete, and will be scheduled back in at a later point.

v2: Handle CONTEXT_STATUS_PREEMPTED in gvt by aliasing it to
CONTEXT_STATUS_OUT for the moment, gvt can expand upon the difference
later.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: "Zhenyu Wang" <zhenyuw@linux.intel.com>
Cc: "Wang, Zhi A" <zhi.a.wang@intel.com>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-3-chris@chris-wilson.co.uk

Not only the context image consist of two parts (the PPHWSP, and the
logical context state), but we also allocate a header at the start of
for sharing data with GuC. Thus every lrc looks like this:

  | [guc]          | [hwsp] [logical state] |
  |<- our header ->|<- context image      ->|

So far, we have oversimplified whenever we use each of these parts of the
context, just because the GuC header happens to be in page 0, and the
(PP)HWSP is in page 1. But this had led to using the same define for more
than one meaning (as a page index in the lrc and as 1 page).

This patch adds defines for the GuC shared page, the PPHWSP page and the
start of the logical state. It also updated the places where the old
define was being used. Since we are not changing the size (or format) of
the context, there are no functional changes.

v2: Use PPHWSP index for hws again.
Suggested-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NMichel Thierry <michel.thierry@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Cc: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Oscar Mateo <oscar.mateo@intel.com>
Cc: intel-gvt-dev@lists.freedesktop.org
Link: http://patchwork.freedesktop.org/patch/msgid/20170712193032.27080-1-michel.thierry@intel.comReviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20170913085605.18299-1-chris@chris-wilson.co.uk

When an error occurs in dispatch_workload, this patch is to do the
proper cleanup and rollback to the original states before the workload
is abandoned.

v2:
- split the mixed several error paths for better review. (Zhenyu)

v3:
- original PTR_ERR(cs) is good and code cleanup. (Zhenyu)

v4:
- reuse the existing i915_add_request for error handling. (Zhenyu)

v5:
- remove the duplicate error handling release_shadow_wa_ctx and
  move the engine->context_unpin upper. (Zhenyu)

v6:
- keep the old label "out". (Zhenyu)
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

When an error occurs after shadow_indirect_ctx, this patch is to do the
proper cleanup and rollback to the original states for shadowed indirect
context before the workload is abandoned.

v2:
- split the mixed several error paths for better review. (Zhenyu)

v3:
- no return check for clean up functions. (Changbin)

v4:
- expose and reuse the existing release_shadow_wa_ctx. (Zhenyu)

v5:
- move the release function to scheduler.c file. (Zhenyu)

v6:
- move error handling code of intel_gvt_scan_and_shadow_workload
  to here. (Zhenyu)
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Currently i915 request structure and shadow ring buffer are allocated
before command scan, so it will have to restore to previous states once
any error happens afterwards in the long dispatch_workload path.

This patch is to introduce a reserved ring buffer created at the beginning
of vGPU initialization. Workload will be coped to this reserved buffer and
be scanned first, the i915 request and shadow ring buffer are only
allocated after the result of scan is successful.

To balance the memory usage and buffer alloc time, the coming bigger ring
buffer will be reallocated and kept until more bigger buffer is coming.

v2:
- use kmalloc for the smaller ring buffer, realloc if required. (Zhenyu)

v3:
- remove the dynamically allocated ring buffer. (Zhenyu)

v4:
- code style polish.
- kfree previous allocated buffer once kmalloc failed. (Zhenyu)
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

The current context logic only updates the descriptor of context when
it's being pinned to graphics memory space. But this cannot satisfy the
requirement of shadow context. The addressing mode of the pinned shadow
context descriptor may be changed according to the guest addressing mode.
And this won't be updated, as the already pinned shadow context has no
chance to update its descriptor. And this will lead to GPU hang issue,
as shadow context is used with wrong descriptor. This patch fixes this
issue by letting the pinned shadow context descriptor update its
addressing mode on demand.

This patch fixes GPU HANG issue which happends after changing the
grub parameter i915.enable_ppgtt form 0x01 to 0x03 or vice versa and
then rebooting the guest.
Signed-off-by: NTina Zhang <tina.zhang@intel.com>
Signed-off-by: NKechen Lu <kechen.lu@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>