Factor out vGPU workload creation/destroy functions since they are not
specific to execlist emulation.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

When a scan error occurs in submit_context, this patch is to
decrease the mm ref count and free the workload struct before
the workload is abandoned.

v2:
- submit_context related code should be combined together. (Zhenyu)

v3:
- free all the unsubmitted workloads. (Zhenyu)

v4:
- refine the clean path. (Zhenyu)

v5:
- polish the title. (Zhenyu)
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Theoretically, the largest bulk of commands in the ring buffer of an
engine might be the first submission, which usually contains a lot
of commands to initialize the HW. After removing the initial allocation
of the ring scan buffer and let krealloc() do everything we need, we
still have a big chance to get the buffer of suitable size in the first
submission.

Tested on my SKL NUC.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Move ring scan buffers into intel_vgpu_submission since they belongs to
a part of vGPU submission stuffs.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

"reserved" means reserve something from somewhere. Actually they are
buffers used by command scanner. Rename it to ring_scan_buffer.

v2:

- Remove the usage of an extra variable. (Zhenyu)

Fixes: 0a53bc07 ("drm/i915/gvt: Separate cmd scan from request allocation")
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Introduce intel_vgpu_submission to hold all members related to submission
in struct intel_vgpu before.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Move vGPU workload cache initialization/de-initialization into
intel_vgpu_{setup, clean}_submission() since they are not specific to
execlist stuffs.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

The context descriptors in elsp_dwords are stored in a reversed order and
the definition of context descriptor is also reversed. The revesred stuff
is hard to be used and might cause misunderstanding. Make them in the right
oder for following code re-factoring.

Tested on my SKL NUC.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Need to check valid state for per_ctx bb and bypass batch buffer
combine for scan if necessary. Otherwise adding invalid MI batch
buffer start cmd for per_ctx bb will cause scan failure, which is
taken as -EFAULT now so vGPU would be put in failsafe. This trys
to fix that by checking per_ctx bb valid state. Also remove old
invalid WARNING that indirect ctx bb shouldn't depend on valid
per_ctx bb.
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

refine the error handling for prepare_execlist_workload to restore to the
original states once error occurs.

only release the shadowed batch buffer and wa ctx when the workload is
completed successfully.

v2:
- split the mixed several error paths for better review. (Zhenyu)

v3:
- handle prepare batch buffer/wa ctx pin errors and
- emulate_schedule_in null issue. (Zhenyu)

v4:
- no need to handle emulate_schedule_in null issue. (Zhenyu)

v5:
- release the shadowed batch buffer and wa ctx only for the
  successful workload. (Zhenyu)

v6:
- polish the return style. (Zhenyu)
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

When an error occurs after shadow_indirect_ctx, this patch is to do the
proper cleanup and rollback to the original states for shadowed indirect
context before the workload is abandoned.

v2:
- split the mixed several error paths for better review. (Zhenyu)

v3:
- no return check for clean up functions. (Changbin)

v4:
- expose and reuse the existing release_shadow_wa_ctx. (Zhenyu)

v5:
- move the release function to scheduler.c file. (Zhenyu)

v6:
- move error handling code of intel_gvt_scan_and_shadow_workload
  to here. (Zhenyu)
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Currently i915 request structure and shadow ring buffer are allocated
before command scan, so it will have to restore to previous states once
any error happens afterwards in the long dispatch_workload path.

This patch is to introduce a reserved ring buffer created at the beginning
of vGPU initialization. Workload will be coped to this reserved buffer and
be scanned first, the i915 request and shadow ring buffer are only
allocated after the result of scan is successful.

To balance the memory usage and buffer alloc time, the coming bigger ring
buffer will be reallocated and kept until more bigger buffer is coming.

v2:
- use kmalloc for the smaller ring buffer, realloc if required. (Zhenyu)

v3:
- remove the dynamically allocated ring buffer. (Zhenyu)

v4:
- code style polish.
- kfree previous allocated buffer once kmalloc failed. (Zhenyu)
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

The current context logic only updates the descriptor of context when
it's being pinned to graphics memory space. But this cannot satisfy the
requirement of shadow context. The addressing mode of the pinned shadow
context descriptor may be changed according to the guest addressing mode.
And this won't be updated, as the already pinned shadow context has no
chance to update its descriptor. And this will lead to GPU hang issue,
as shadow context is used with wrong descriptor. This patch fixes this
issue by letting the pinned shadow context descriptor update its
addressing mode on demand.

This patch fixes GPU HANG issue which happends after changing the
grub parameter i915.enable_ppgtt form 0x01 to 0x03 or vice versa and
then rebooting the guest.
Signed-off-by: NTina Zhang <tina.zhang@intel.com>
Signed-off-by: NKechen Lu <kechen.lu@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Need to take runtime pm when do early scan/shadow of workload
for request operations.

Fixes: 7fa56bd159bc ("drm/i915/gvt: Audit and shadow workload during ELSP writing")
Cc: Ping Gao <ping.a.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Let the workload audit and shadow ahead of vGPU scheduling, that
will eliminate GPU idle time and improve performance for multi-VM.

The performance of Heaven running simultaneously in 3VMs has
improved 20% after this patch.

v2:Remove condition current->vgpu==vgpu when shadow during ELSP
writing.
Signed-off-by: NPing Gao <ping.a.gao@intel.com>
Reviewed-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

If a workload caused a HW GPU hang or it is in the middle of
vGPU reset, the workload queue should be cleaned up to emulate
the hang state of the GPU.

v2:
- use ENGINE_MASK(ring_id) instead of (1 << ring_id). (Zhenyu)
Signed-off-by: NChuanxiao Dong <chuanxiao.dong@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Use resetting_eng to identify which engine is resetting
so the rest ones' workload won't be impacted

v2:
- use ENGINE_MASK(ring_id) instead of (1 << ring_id). (Zhenyu)
Signed-off-by: NChuanxiao Dong <chuanxiao.dong@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

The function intel_vgpu_submit_execlist could be more simpler. It
actually does:
  1) validate the submission. The first context must be valid,
     and all two must be privilege_access.
  2) submit valid contexts. The first one need emulate schedule_in.

We do not need a bitmap, valid desc copy valid_desc. Local variable
emulate_schedule_in also can be optimized out.

v2: dump desc content in err msg (Zhi Wang)
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Reviewed-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

This is to fix a memory leak issue caused by unfreed gvtg workload objects.
Walk through the workload list and free all of the remained workloads
before destroying kmem cache.

[179.885211] INFO: Object 0xffff9cef10003b80 @offset=7040
[179.885657] kmem_cache_destroy gvt-g_vgpu_workload: Slab cache still has objects
[179.886146] CPU: 2 PID: 2318 Comm: win_lucas Tainted: G    B   W       4.11.0+ #1
[179.887223] Call Trace:
[179.887394]  dump_stack+0x63/0x90
[179.887617]  kmem_cache_destroy+0x1cf/0x1e0
[179.887960]  intel_vgpu_clean_execlist+0x15/0x20 [i915]
[179.888365]  intel_gvt_destroy_vgpu+0x4c/0xd0 [i915]
[179.888688]  intel_vgpu_remove+0x2a/0x30 [kvmgt]
[179.888988]  mdev_device_remove_ops+0x23/0x50 [mdev]
[179.889309]  mdev_device_remove+0xe4/0x190 [mdev]
[179.889615]  remove_store+0x7d/0xb0 [mdev]
[179.889885]  dev_attr_store+0x18/0x30
[179.890129]  sysfs_kf_write+0x37/0x40
[179.890371]  kernfs_fop_write+0x107/0x180
[179.890632]  __vfs_write+0x37/0x160
[179.890865]  ? kmem_cache_alloc+0xd7/0x1b0
[179.891116]  ? apparmor_file_permission+0x1a/0x20
[179.891372]  ? security_file_permission+0x3b/0xc0
[179.891628]  vfs_write+0xb8/0x1b0
[179.891812]  SyS_write+0x55/0xc0
[179.891992]  entry_SYSCALL_64_fastpath+0x1e/0xad
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

There are two bugs here.  The && should be || and the > is off by one so
it should be >= ARRAY_SIZE().

Fixes: 8453d674 ("drm/i915/gvt: vGPU execlist virtualization")
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Let c compiler handle the structure copying. The compiler will use
builtin function to handle that.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Fix wrong initial csb read pointer value. This fixes the random
engine timeout issue in guest when guest boots up.

Fixes: 8453d674 ("drm/i915/gvt: vGPU execlist virtualization")
Cc: stable@vger.kernel.org # v4.10+
Signed-off-by: NMin He <min.he@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

intel_shadow_wa_ctx is a field of intel_vgpu_workload. container_of() can
be used to refine the relation-ship between intel_shadow_wa_ctx and
intel_vgpu_workload. This patch removes the useless dereference.

v2. add "drm/i915/gvt" prefix. (Zhenyu)
Signed-off-by: NTina Zhang <tina.zhang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

gvt_err should be used only for the very few critical error message
during host i915 drvier initialization. This patch
1. removes the redundant gvt_err;
2. creates a new gvt_vgpu_err to show errors caused by vgpu;
3. replaces the most gvt_err with gvt_vgpu_err;
4. leaves very few gvt_err for dumping gvt error during host gvt
   initialization.

v2. change name to gvt_vgpu_err and add vgpu id to the message. (Kevin)
    add gpu id to gvt_vgpu_err. (Zhi)
v3. remove gpu id from gvt_vgpu_err caller. (Zhi)
v4. add vgpu check to the gvt_vgpu_err macro. (Zhiyuan)
v5. add comments for v3 and v4.
v6. split the big patch into two, with this patch only for checking
    gvt_vgpu_err. (Zhenyu)
v7. rebase to staging branch
v8. rebase to fix branch
Signed-off-by: NTina Zhang <tina.zhang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

In function dispatch_workload(), if it fail before calling
intel_gvt_scan_and_shadow_wa_ctx(), the indirect ctx will
not be shadowed so no cleaup need. wa_ctx->indirect_ctx.obj
indicate whether indirect_ctx is shadowed. The obj is null
if it is unshadowed.

BUG: unable to handle kernel NULL pointer dereference at
00000000000001a0
IP: complete_execlist_workload+0x2c9/0x3e0 [i915]
Oops: 0002 [#1] SMP
task: ffff939546d2d880 task.stack: ffffbd9b82ac4000
RIP: 0010:complete_execlist_workload+0x2c9/0x3e0 [i915]
RSP: 0018:ffffbd9b82ac7dd8 EFLAGS: 00010202
RAX: 0000000000000000 RBX: ffff9393c725b540 RCX: 0000000000000006
RDX: 0000000000000007 RSI: 0000000000000202 RDI: ffff939559c8dd00
RBP: ffffbd9b82ac7e18 R08: 0000000000000001 R09: 000000000120dd8f
R10: 0000000000000000 R11: 000000000120dd8f R12: ffff9393c725b540
R13: ffff9393c725b618 R14: ffffbd9b81f0d000 R15: ffff939520e0e000
FS:  0000000000000000(0000) GS:ffff939559c80000(0000)
knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001a0 CR3: 000000043d664000 CR4: 00000000003426e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 workload_thread+0x312/0xd70 [i915]
 ? __wake_up_sync+0x20/0x20
 ? wake_atomic_t_function+0x60/0x60
 kthread+0x101/0x140
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

According to kmem_cache_sanity_check(), spaces are not allowed in the
name of a cache and results in a kernel oops with CONFIG_DEBUG_VM.
Convert to underscores.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

set_gma_to_bb_cmd() is completely bogus - it is (incorrectly) applying
the rules to read a GTT offset from a command as opposed to writing the
GTT offset. And to cap it all set_gma_to_bb_cmd() is called within a list
iterator of the most strange construction.

Fixes: be1da707 ("drm/i915/gvt: vGPU command scanner")
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Cc: Yulei Zhang <yulei.zhang@intel.com>
Cc: <drm-intel-fixes@lists.freedesktop.org> # v4.10-rc1+
Tested-by: NTina Zhang <tina.zhang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

We initiate vgpu->workload_q_head via for_each_engine
macro which may skip unavailable engines. So we should
follow this rule anywhere. The function
intel_vgpu_reset_execlist is not aware of this. Kernel
crash when touch a uninitiated vgpu->workload_q_head[x].
Let's fix it by using for_each_engine_masked and skip
unavailable engine ID. Meanwhile rename ring_bitmap to
general name engine_mask.

v2: remove unnecessary engine activation check (zhenyu)
Signed-off-by: NDu, Changbin <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

The function return values should has type int if it return
a integer value.
Signed-off-by: NDu, Changbin <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Mark all local functions & variables as static.
Signed-off-by: NDu, Changbin <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Switch to use new for_each_engine() helper to properly access
enabled intel_engine_cs as i915 core has changed that to be
dynamic managed. At GVT-g init time would still depend on ring
mask to determine engine list as it's earlier.
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

We have the ability to map an object, so use it rather than opencode it
badly. Note that the object remains permanently pinned, this is poor
practise.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

We have the ability to map an object, so use it rather than opencode it
badly.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Unpinning the pages prior to the object being release from the GPU may
allow the GPU to read and write into system pages (i.e. use after free
by the hw).
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

The purpose of returning the just-pinned VMA is so that we can use the
information within, like its address. Also it should be tracked and used
as the cookie to unpin...
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NZhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Don't use obsolete drm_gem_object_unreference() but switch to i915_gem_object_put().
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

i915 core should only call functions and structures exposed through
intel_gvt.h. Remove internal gvt.h and i915_pvinfo.h.

Change for internal intel_gvt structure as private handler which
not requires to expose gvt internal structure for i915 core.

v2: Fix per Chris's comment
- carefully handle dev_priv->gvt assignment
- add necessary bracket for macro helper
- forward declartion struct intel_gvt
- keep free operation within same file handling alloc

v3: fix use after free and remove intel_gvt.initialized

v4: change to_gvt() to an inline
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Change GVT-g code reference for intel_engine_cs from static array to
allocated pointer after commit 3b3f1650 ("drm/i915: Allocate
intel_engine_cs structure only for the enabled engines").
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: http://patchwork.freedesktop.org/patch/msgid/20161018014007.29369-1-zhenyuw@linux.intel.com

This patch introduces a command scanner to scan guest command buffers.
Signed-off-by: NYulei Zhang <yulei.zhang@intel.com>
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

This patch introduces the vGPU workload scheduler routines.

GVT workload scheduler is responsible for picking and executing GVT workload
from current scheduled vGPU. Before the workload is submitted to host i915,
the guest execlist context will be shadowed in the host GVT shadow context.
the instructions in guest ring buffer will be copied into GVT shadow ring
buffer. Then GVT-g workload scheduler will scan the instructions in guest
ring buffer and submit it to host i915.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>