Accurate names help to avoid confusing so improve readability.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Reviewed-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

This add a new macro gvt_vdbg_mm() to print more verbose logs for
gtt shadowing. The added verbose logs are very useful for debugging.
gvt_vdbg_mm() only comes into effect if VERBOSE_DEBUG is defined by
the developer.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Reviewed-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Less code and use existed helper ggtt_set_host_entry.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Separate ggtt and ppgtt since they are different. A little more code but
straightforward.

And move these helpers to gtt.c since that is the only client.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

If we manage an object with a reference count, then its life cycle
must flow the reference count operations. Meanwhile, change the
operation functions to generic name *put* and *get*.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

This is a big one and the GVT shadow graphic memory management code is
heavily refined. The new code is more straightforward with less code.

The struct intel_vgpu_mm is restructured to be clearly defined, use
accurate names and some of the original fields are removed which are
really redundant.

Now we only manage ppgtt mm object with mm->ppgtt_mm.lru_list. No need
to mix ppgtt and ggtt together, since one vGPU only has one ggtt object.

v4: Don't invoke ppgtt_free_all_shadow_page before intel_vgpu_destroy_all_ppgtt_mm.
v3: Add GVT_RING_CTX_NR_PDPS to avoid confusing about the PDPs.
v2: Split some changes into small standalone patches.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

GVT may receive partial write on one guest PTE update. Validate gfn
not to translate incomplete gfn. This avoids some unnecessary error
messages incurred by the incomplete gfn translating. Also fix the
bug that the whole PPGTT shadow page update is aborted on any invalid
gfn entry.

gfn validation relys on hypervisor's help. Add one MPT module function
to provide the function.
Signed-off-by: NHang Yuan <hang.yuan@intel.com>
Reviewed-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NRodrigo Vivi <rodrigo.vivi@intel.com>

A shadow page table entry needs to be cleared after being set as
post-sync. This patch fixes the recent error reported in Win7-32 test.

Fixes: 2707e444 ("drm/i915/gvt: vGPU graphics memory virtualization")
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>
CC: Stable <stable@vger.kernel.org>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

It's a bit confusing that page write protect handler is live in
mmio emulation handler. This moves it to stand alone gvt ops.

Also remove unnecessary check of write protected page access
in mmio read handler and cleanup handling of failsafe case.

v2: rebase
Reviewed-by: NXiong Zhang <xiong.y.zhang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

We had previous hack that tried to accept either i915_reg_t or offset
value to access vGPU virtual/shadow regs which broke that purpose to
be type safe in context. This one trys to explicitly separate the usage
of typed mmio reg with real offset.

Old vgpu_vreg(offset) helper is used only for offset now with new
vgpu_vreg_t(reg) is used for i915_reg_t only. Convert left usage
of that to new helper.

Also fixed left KASAN warning issues caused by previous hack.

v2: rebase, fixup against recent mmio switch change
Reviewed-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

For ADDR_4K_MASK, bit[45..12] should be 1, all other bits
should be 0. The current definition wrongly set bit[46] as 1
also. This path fixes this.

v2: Add commit message, fixes and cc stable.(Zhenyu)

Fixes: 2707e444("drm/i915/gvt: vGPU graphics memory virtualization")
Signed-off-by: NXiong Zhang <xiong.y.zhang@intel.com>
Cc: stable@vger.kernel.org
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

This reverts commit b20d09886fd1b74cd2255d846029a049e524db14.

This caused windows driver boot errors for invalid page address.
Revert for now.
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Refine previously broken PPGTT scratch. Scratch PTE was no correctly
handled and also the handling of scratch entries in page table walk was
not well organized, which brings gaps of introducing lazy shadow.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

We need ops->set_present() during generating a new scratch page table
entry.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Need to figure out page table type of current level by GTT entry type
during getting a scratch page table entry.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

During a vGPU reset, the scratch page table shouldn't be cleared, what
needs to be cleared should be the scratch page.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

As we want to re-use intel_vgpu_shadow_page in buidling scrach page table
and we don't want to put scrach page table page into hash table, a new
param is introduced to give the caller a choice to decide if a shadow page
should be put into hash table.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

As there is already an I915_GTT_PAGE_SIZE marco in i915, let GVT-g use it
as well. Also this patch re-names some GTT marcos with additional prefix.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

As the data structure of "intel_vgpu_guest_page" will become much heavier
in future, it's better to factor out the guest memory page track mechnisim
as early as possible.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Replace the plain bit usage with BIT() to make klockwork happy.

Cc: Deng Hongyi <hongyi.deng@intel.com>
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>

Remove the "INDEX" suffix from PPAT marcos as they are bits actually, not
indexes.
Suggested-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NJoonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/1505392783-4084-2-git-send-email-zhi.a.wang@intel.com

When it is failed in shadow_mm, the pin_count should rollback
to the original states before return.

v2:
- split the mixed several error paths for better review. (Zhenyu)

v3:
  increase the pincount after shadow success. (Zhenyu)
Signed-off-by: Nfred gao <fred.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Guest i915 full ppgtt functionality was blocking by an issue, which would
lead to gpu hardware hang. Guest i915 driver may update the ppgtt table
just before this workload is going to be submitted to the hardware by
device model. This case wasn't handled well by device model before, due
to the small time window between removing old ppgtt entry and adding the
new one. Errors occur when the workload is executed by hardware during
that small time window. This patch is to remove this time window by adding
the new ppgtt entry first and then remove the old one.

Changes in v2:
- Move VGT_CAPS_FULL_PPGTT introduction to patch 2/4. (Joonas)

Changes since v2:
- Divide the whole patch set into two separate patch series, with one
  patch in i915 side to check guest i915 full ppgtt capability and enable
  it when this capability is supported by the device model, and the other
  one in gvt side which fixs the blocking issue and enables the device
  model to provide the capability to guest. And this patch focuses on gvt
  side. (Joonas)
- Change the title from "reorder the shadow ppgtt update process by adding
  entry first" to "Fix guest i915 full ppgtt blocking issue". (Tina)

Changes since v3:
- Rebase to the latest branch.

Changes since v4:
- Tested by Tina Zhang.

Changes since v5:
- Rebase to the latest branch.

v6:
- Update full 48bit ppgtt definition

Cc: Tina Zhang <tina.zhang@intel.com>
Signed-off-by: NTina Zhang <tina.zhang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

When doing the VGPU reset, we don't need to do the gtt/ppgtt reset.
This will make the GVT to do the ppgtt shadow every time for
a workload and caused really bad performance after a VGPU reset.
This patch will make sure ppgtt clean only happen at device module
level reset to fix this.
Signed-off-by: NChuanxiao Dong <chuanxiao.dong@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

When debugging the gtt code, found the intel_vgpu_gma_to_gpa() can
translate any given GMA though the GMA is not valid. This because
the GTT ops suppress the possible errors, which may result in an
invalid PT entry is retrieved by upper caller.

This patch changed the prototype of pte ops to propagate status to
callers. Then we make sure the GTT walker stop as early as when
a error is detected to prevent undefined behavior.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

It will causes memory leak, if the function setup_spt_oos() fail,
in the function intel_gvt_init_gtt(),
which allocated by get_zeroed_page() and mapped by dma_map_page().

Unmap and free the page,  after STP oos initialize fail,
it will fix this issue.
Signed-off-by: NZhou, Wenjia <zhiyuan_zhu@htc.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

add gtt_invalidate API to handle the GTT TLB flush instead of
hiding in write_pte64 function. This can avoid overkill when using
write_pte64
Suggested-by: NZhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NChuanxiao Dong <chuanxiao.dong@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Make sure to take runtime pm when write PTE flush which ensure to
write to hw properly. This fixes warning during mdev/vgpu creation
which will do ggtt reset.

------------[ cut here ]------------
WARNING: CPU: 1 PID: 9375 at drivers/gpu/drm/i915/intel_drv.h:1748 fwtable_write32+0x1c2/0x1e0 [i915]
 RPM wakelock ref not held during HW access
Call Trace:
  ? dump_stack+0x5c/0x81
  ? __warn+0xbe/0xe0
  ? warn_slowpath_fmt+0x5a/0x80
  ? wake_up_klogd+0x37/0x40
  ? vprintk_emit+0x2ef/0x370
  ? fwtable_write32+0x1c2/0x1e0 [i915]
  ? gtt_set_entry64+0xbb/0xd0 [i915]
  ? intel_vgpu_reset_ggtt+0x88/0xf0 [i915]
  ? intel_vgpu_init_gtt+0xa5/0x4f0 [i915]
  ? intel_gvt_create_vgpu+0x1b5/0x250 [i915]
  ? kobject_put+0x1b/0x50
  ? intel_vgpu_create+0x4e/0x130 [kvmgt]
  ? mdev_device_create+0x186/0x2a0 [mdev]
  ? create_store+0xba/0xe0 [mdev]
  ? create_store+0xba/0xe0 [mdev]
  ? kernfs_fop_write+0x109/0x1a0
  ? kernfs_fop_write+0x109/0x1a0
  ? __vfs_write+0x33/0x160
  ? __fput+0x161/0x1d0
  ? vfs_write+0xb0/0x190
  ? SyS_write+0x52/0xc0
  ? exit_to_usermode_loop+0x7a/0xa0
  ? entry_SYSCALL_64_fastpath+0x1e/0xad

v2: remove unrelated oops info

v3: change to take runtime pm for ggtt reset instead of get/put for
    each pte write flush

Fixes: d650ac06 ("drm/i915/gvt: reset the GGTT entry when vGPU created")
Cc: Ping Gao <ping.a.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Extend function dispatch logic to support KBL platform.
Signed-off-by: NXu Han <xu.han@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Sometimes guest driver will only update partial of the GGTT entry then
access it. In this situation a failure will happen while translating
the gpa to hpa.
Now in this situation we let the corresponding shadow entry pointing
to a scratch page.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NXiaoguang Chen <xiaoguang.chen@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

gvt_err should be used only for the very few critical error message
during host i915 drvier initialization. This patch
1. removes the redundant gvt_err;
2. creates a new gvt_vgpu_err to show errors caused by vgpu;
3. replaces the most gvt_err with gvt_vgpu_err;
4. leaves very few gvt_err for dumping gvt error during host gvt
   initialization.

v2. change name to gvt_vgpu_err and add vgpu id to the message. (Kevin)
    add gpu id to gvt_vgpu_err. (Zhi)
v3. remove gpu id from gvt_vgpu_err caller. (Zhi)
v4. add vgpu check to the gvt_vgpu_err macro. (Zhiyuan)
v5. add comments for v3 and v4.
v6. split the big patch into two, with this patch only for checking
    gvt_vgpu_err. (Zhenyu)
v7. rebase to staging branch
v8. rebase to fix branch
Signed-off-by: NTina Zhang <tina.zhang@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

The guest VM may initialize the whole GTT table during boot up,
so the warning msg in emulate_gtt_mmio_write is not necessary, it is
the expected behavior and it may confuse the user if error msg is
printed out, so remove the msg from emulate_gtt_mmio_write(),
Signed-off-by: NZhao, Xinda <xinda.zhao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

Releasing shadow PPGTT pages is not enough when vGPU reset, the
guest page table tracking data should has same life-cycle with
all the shadow PPGTT pages; Otherwise there is no chance to
re-shadow the PPGTT pages without free the guest page table
tracking data.

This patch clear the PPGTT reset logic and make the vGPU reset in
working order.

v2: refactor some logic to avoid code duplicated.
v3: remove useless macro and add comments from Christophe.
v4: keep reset logic in reset function.
Signed-off-by: NPing Gao <ping.a.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

MFN usually refers to "Machine Frame Number" in virtulization world.
Currently GVT-g populates the shadow PPGTT/GGTT page table with MFN
according to the translation between MFN and Guest PFN.

When host IOMMU is enabled, GPU DMA transactions go through the IOMMU,
GPU needs an IOVA<->MFN mapping to walk the shadow page table in host
main memory.

This patch will map a page in IOMMU page table before using it in shadow
page table and release the map when a shadow page is going to be freed.
Signed-off-by: NZhi Wang <zhi.a.wang@intel.com>
Signed-off-by: NChuanxiao Dong <chuanxiao.dong@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

This patch introduces a new function intel_vgpu_reset_gtt() to reset
the all GTT related status, including GGTT, PPGTT, scratch page. This
function can free all shadowed PPGTT, clear all GGTT entry, and clear
scratch page to all zero. After this, we can ensure no gtt related
information can be leakaged from one VM to anothor one when assign
vgpu instance across different VMs (not simultaneously).
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

In gvt, almost all memory allocations are in sleepable contexts. It's
fault-prone to use GFP_ATOMIC everywhere. Replace it with GFP_KERNEL
wherever possible.
Signed-off-by: NJike Song <jike.song@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

The readq and writeq are already offered by drm_os_linux.h. So we can
use them directly whithout dectecting their presence. This patch removed
the duplicated code.
Signed-off-by: NChangbin Du <changbin.du@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

The GGTT space is partitioned between vGPUs, it could be reused by
next vGPU after previous one is release, the stale entries need
point to scratch page when vGPU created.

v2: Reset logic move to vGPU create.
v3: Correct the commit msg.
v4: Move the reset function to vGPU init gtt function, as result it's no
    need explicitly in vGPU reset logic as vGPU init gtt called during
    reset.
Signed-off-by: NPing Gao <ping.a.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

On guest writing a PPGTT entry, if it contains value and the old
entry is valid, gvt will read it and find & free the corresponding
old data for it. However, with the KVM write protection provided
by page_track, the guest entry will be written with new value
before gvt handling. To avoid that, we should use the shadow
entry instead.
Signed-off-by: NBing Niu <bing.niu@intel.com>
Signed-off-by: NJike Song <jike.song@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>

All the unused entries in the page table tree(PML4E->PDPE->PDE->PTE)
should point to scratch page table/scratch page to avoid page walk error
due to the page prefetching.
When removing an entry in shadow PPGTT,  it need map to scratch page
also, the older implementation use single scratch page to assign to all
level entries, it doesn't align the page walk behavior when removed
entry is in PML, PDP, PD.  To avoid potential page walk error this patch
implement a scratch page tree to replace the single scratch page.

v2: more details in commit message address Kevin's comments.
Signed-off-by: NPing Gao <ping.a.gao@intel.com>
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>