Move fences that have already signaled should not prevent memory
allocations with no_wait_gpu.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210227034524.21763-1-Felix.Kuehling@amd.comSigned-off-by: NChristian König <christian.koenig@amd.com>

The BO might be NULL in this function, use the bdev directly.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reported-by: NColin Ian King <colin.king@canonical.com>
Reviewed-by: NColin Ian King <colin.king@canonical.com>
Fixes: a1f091f8 ("drm/ttm: switch to per device LRU lock")
Link: https://patchwork.freedesktop.org/patch/msgid/20210325152740.82633-1-christian.koenig@amd.com

Instead of having a global lock for potentially less contention.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Tested-by: NNirmoy Das <nirmoy.das@amd.com>
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NMatthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/424010/

Instead evict round robin from each devices SYSTEM and TT domain.

v2: reorder num_pages access reported by Dan's script
v3: fix rebase fallout, num_pages should be 32bit
Signed-off-by: NChristian König <christian.koenig@amd.com>
Tested-by: NNirmoy Das <nirmoy.das@amd.com>
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NMatthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/424009/

Move the iteration of the global lru into the new function
ttm_global_swapout() and use that instead in drivers.

v2: consistently return int
v3: fix build fail
Signed-off-by: NChristian König <christian.koenig@amd.com>
Tested-by: NNirmoy Das <nirmoy.das@amd.com>
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NMatthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/424008/

QXL indeed unrefs pinned BOs and the warnings are spamming peoples log files.

Make sure we warn only once until the QXL driver is fixed.
Signed-off-by: NChristian König <christian.koenig@amd.com>
References: https://lore.kernel.org/lkml/YD+eYcMMcdlXB8PY@alley/
Link: https://patchwork.freedesktop.org/patch/422834/Reviewed-by: NDaniel Vetter <daniel@ffwll.ch>
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@linux.intel.com>

BO would be added into swap list if it is validated into system domain.
If BO is validated again into non-system domain, say, VRAM domain. It
actually should not be in the swap list.
Signed-off-by: Nxinhui pan <xinhui.pan@amd.com>
Acked-by: NGuchun Chen <guchun.chen@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210224032808.150465-1-xinhui.pan@amd.comSigned-off-by: NChristian König <christian.koenig@amd.com>

Free the memory on failure.
Also no need to re-alloc memory on retry.
Signed-off-by: Nxinhui pan <xinhui.pan@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210219042547.44855-1-xinhui.pan@amd.comReviewed-by: NChristian König <christian.koenig@amd.com>
CC: stable@vger.kernel.org # 5.11
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@linux.intel.com>

This is just another feature which is only used by VMWGFX, so move
it into the driver instead.

I've tried to add the accounting sysfs file to the kobject of the drm
minor, but I'm not 100% sure if this works as expected.

v2: fix typo in KFD and avoid 64bit divide
v3: fix init order in VMWGFX
v4: use pdev sysfs reference instead of drm
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: Zack Rusin <zackr@vmware.com> (v3)
Tested-by: NNirmoy Das <nirmoy.das@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210208133226.36955-2-christian.koenig@amd.com

TTM implements a rather extensive accounting of allocated memory.

There are two reasons for this:
1. It tries to block userspace allocating a huge number of very small
   BOs without accounting for the kmalloced memory.

2. Make sure we don't over allocate and run into an OOM situation
   during swapout while trying to handle the memory shortage.

This is only partially a good idea. First of all it is perfectly
valid for an application to use all of system memory, limiting it to
50% is not really acceptable.

What we need to take care of is that the application is held
accountable for the memory it allocated. This is what control
mechanisms like memcg and the normal Linux page accounting already do.

Making sure that we don't run into an OOM situation while trying to
cope with a memory shortage is still a good idea, but this is also
not very well implemented since it means another opportunity of
recursion from the driver back into TTM.

So start to rework all of this by implementing a shrinker callback which
allows for TT object to be swapped out if necessary.

v2: Switch from limit to shrinker callback.
v3: fix gfp mask handling, use atomic for swapable_pages, add debugfs
v4: drop the extra gfp_mask checks
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20210208133226.36955-1-christian.koenig@amd.com

Rename ttm_bo_device to ttm_device.
Rename ttm_bo_driver to ttm_device_funcs.
Rename ttm_bo_global to ttm_global.

Move global and device related functions to ttm_device.[ch].

No functional change.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/415222/

As far as I can tell the buffer_count was never used by an
userspace application.

The number of BOs in the system is far better suited in
debugfs than sysfs and we now should be able to add other
information here as well.

v2: add that additionally to sysfs
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/414954/

Recently a regression was introduced which caused TTM's buffer eviction to
attempt to evict already-pinned BOs, causing issues with buffer eviction
under memory pressure along with suspend/resume:

  nouveau 0000:1f:00.0: DRM: evicting buffers...
  nouveau 0000:1f:00.0: DRM: Moving pinned object 00000000c428c3ff!
  nouveau 0000:1f:00.0: fifo: fault 00 [READ] at 0000000000200000 engine 04
  [BAR1] client 07 [HUB/HOST_CPU] reason 02 [PTE] on channel -1 [00ffeaa000
  unknown]
  nouveau 0000:1f:00.0: fifo: DROPPED_MMU_FAULT 00001000
  nouveau 0000:1f:00.0: fifo: fault 01 [WRITE] at 0000000000020000 engine
  0c [HOST6] client 07 [HUB/HOST_CPU] reason 02 [PTE] on channel 1
  [00ffb28000 DRM]
  nouveau 0000:1f:00.0: fifo: channel 1: killed
  nouveau 0000:1f:00.0: fifo: runlist 0: scheduled for recovery
  [TTM] Buffer eviction failed
  nouveau 0000:1f:00.0: DRM: waiting for kernel channels to go idle...
  nouveau 0000:1f:00.0: DRM: failed to idle channel 1 [DRM]
  nouveau 0000:1f:00.0: DRM: resuming display...

After some bisection and investigation, it appears this resulted from the
recent changes to ttm_bo_move_to_lru_tail(). Previously when a buffer was
pinned, the buffer would be removed from the LRU once ttm_bo_unreserve
to maintain the LRU list when pinning or unpinning BOs. However, since:

commit 3d1a88e1 ("drm/ttm: cleanup LRU handling further")

We've been exiting from ttm_bo_move_to_lru_tail() at the very beginning of
the function if the bo we're looking at is pinned, resulting in the pinned
BO never getting removed from the lru and as a result - causing issues when
it eventually becomes time for eviction.

So, let's fix this by calling ttm_bo_del_from_lru() from
ttm_bo_move_to_lru_tail() in the event that we're dealing with a pinned
buffer.

v2 (chk): reduce to only the fixing one liner since we always want to
call the callback whenever we would move on the LRU.

Fixes: 3d1a88e1 ("drm/ttm: cleanup LRU handling further")
Cc: Dave Airlie <airlied@redhat.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NLyude Paul <lyude@redhat.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210105114505.38210-1-christian.koenig@amd.com

We only completely delete the BO from the LRU on destruction.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NAndrey Grodzovsky <andrey.grodzovsky@amd.com>
Link: https://patchwork.freedesktop.org/patch/404618/

Check the pin_count instead of the lru list is empty here.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NAndrey Grodzovsky <andrey.grodzovsky@amd.com>
Link: https://patchwork.freedesktop.org/patch/404617/

Based on an idea from Dave, but cleaned up a bit.

We had multiple fields for essentially the same thing.

Now bo->base.size is the original size of the BO in
arbitrary units, usually bytes.

bo->mem.num_pages is the size in number of pages in the
resource domain of bo->mem.mem_type.

v2: use the GEM object size instead of the BO size
v3: fix printks in some places
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com> (v1)
Acked-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/406831/

ttm_module.h deals with internals of TTM and should never
be include outside of it.

v2: also move the file around
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/404885/

Not technically a problem for ttm, but very likely a driver bug and
pretty big time confusing for reviewing code.

So warn about it, both at cleanup time (so we catch these for sure)
and at pin/unpin time (so we know who's the culprit).
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@intel.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201028113120.3641237-1-daniel.vetter@ffwll.ch

Fixes the following W=1 kernel build warning(s):

 drivers/gpu/drm/ttm/ttm_bo.c:51: warning: Function parameter or member 'ttm_global_mutex' not described in 'DEFINE_MUTEX'
 drivers/gpu/drm/ttm/ttm_bo.c:286: warning: Function parameter or member 'bo' not described in 'ttm_bo_cleanup_memtype_use'
 drivers/gpu/drm/ttm/ttm_bo.c:359: warning: Function parameter or member 'bo' not described in 'ttm_bo_cleanup_refs'
 drivers/gpu/drm/ttm/ttm_bo.c:359: warning: Function parameter or member 'interruptible' not described in 'ttm_bo_cleanup_refs'
 drivers/gpu/drm/ttm/ttm_bo.c:359: warning: Function parameter or member 'no_wait_gpu' not described in 'ttm_bo_cleanup_refs'
 drivers/gpu/drm/ttm/ttm_bo.c:359: warning: Function parameter or member 'unlock_resv' not described in 'ttm_bo_cleanup_refs'
 drivers/gpu/drm/ttm/ttm_bo.c:424: warning: Function parameter or member 'bdev' not described in 'ttm_bo_delayed_delete'
 drivers/gpu/drm/ttm/ttm_bo.c:424: warning: Function parameter or member 'remove_all' not described in 'ttm_bo_delayed_delete'
 drivers/gpu/drm/ttm/ttm_bo.c:635: warning: Function parameter or member 'bo' not described in 'ttm_bo_evict_swapout_allowable'
 drivers/gpu/drm/ttm/ttm_bo.c:635: warning: Function parameter or member 'ctx' not described in 'ttm_bo_evict_swapout_allowable'
 drivers/gpu/drm/ttm/ttm_bo.c:635: warning: Function parameter or member 'locked' not described in 'ttm_bo_evict_swapout_allowable'
 drivers/gpu/drm/ttm/ttm_bo.c:635: warning: Function parameter or member 'busy' not described in 'ttm_bo_evict_swapout_allowable'
 drivers/gpu/drm/ttm/ttm_bo.c:769: warning: Function parameter or member 'bo' not described in 'ttm_bo_add_move_fence'
 drivers/gpu/drm/ttm/ttm_bo.c:769: warning: Function parameter or member 'man' not described in 'ttm_bo_add_move_fence'
 drivers/gpu/drm/ttm/ttm_bo.c:769: warning: Function parameter or member 'mem' not described in 'ttm_bo_add_move_fence'
 drivers/gpu/drm/ttm/ttm_bo.c:769: warning: Function parameter or member 'no_wait_gpu' not described in 'ttm_bo_add_move_fence'
 drivers/gpu/drm/ttm/ttm_bo.c:806: warning: Function parameter or member 'bo' not described in 'ttm_bo_mem_force_space'
 drivers/gpu/drm/ttm/ttm_bo.c:806: warning: Function parameter or member 'place' not described in 'ttm_bo_mem_force_space'
 drivers/gpu/drm/ttm/ttm_bo.c:806: warning: Function parameter or member 'mem' not described in 'ttm_bo_mem_force_space'
 drivers/gpu/drm/ttm/ttm_bo.c:806: warning: Function parameter or member 'ctx' not described in 'ttm_bo_mem_force_space'
 drivers/gpu/drm/ttm/ttm_bo.c:872: warning: Function parameter or member 'bo' not described in 'ttm_bo_mem_space'
 drivers/gpu/drm/ttm/ttm_bo.c:872: warning: Function parameter or member 'placement' not described in 'ttm_bo_mem_space'
 drivers/gpu/drm/ttm/ttm_bo.c:872: warning: Function parameter or member 'mem' not described in 'ttm_bo_mem_space'
 drivers/gpu/drm/ttm/ttm_bo.c:872: warning: Function parameter or member 'ctx' not described in 'ttm_bo_mem_space'
 drivers/gpu/drm/ttm/ttm_bo.c:1387: warning: Function parameter or member 'ctx' not described in 'ttm_bo_swapout'
Reviewed-by: NChristian König <christian.koenig@amd.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Signed-off-by: NLee Jones <lee.jones@linaro.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20201116174112.1833368-32-lee.jones@linaro.org

Currently drivers get called to move a buffer, but if they have to
move it temporarily through another space (SYSTEM->VRAM via TT)
then they can end up with a lot of ttm->driver->ttm call stacks,
if the temprorary space moves requires eviction.

Instead of letting the driver do all the placement/space for the
temporary, allow it to report back (-EMULTIHOP) and a placement (hop)
to the move code, which will then do the temporary move, and the
correct placement move afterwards.

This removes a lot of code from drivers, at the expense of
adding some midlayering. I've some further ideas on how to turn
it inside out, but I think this is a good solution to the call
stack problems.

v2: separate out the driver patches, add WARN for getting
MULTHOP in paths we shouldn't (Daniel)
v3: use memset (Christian)
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Nhristian König <christian.koenig@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201109005432.861936-2-airlied@gmail.com

The ttm_operation_ctx structure has a mixture of flags and bools. Drop the
flags and replace them with bools as well.

v2: fix typos, improve comments
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/398686/

Not used any more.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NDave Airlie <airlied@redhat.com>
Reviewed-by: NMadhav Chauhan <madhav.chauhan@amd.com>
Tested-by: NHuang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/397087/?series=83051&rev=1

Provide the necessary parameters by all drivers and use the new pool alloc
when no driver specific function is provided.

v2: fix the GEM VRAM helpers
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NDave Airlie <airlied@redhat.com>
Reviewed-by: NMadhav Chauhan <madhav.chauhan@amd.com>
Tested-by: NHuang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/397081/?series=83051&rev=1

It makes no difference to kmalloc if the structure
is 48 or 64 bytes in size.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NDave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/396950/

The move notify callback is only used in one place, this should
be removed in the future, but for now just rename it to the use
case which is to notify the driver that the GPU memory is to be
deleted.

Drivers can be cleaned up after this separately.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201021044031.1752624-2-airlied@gmail.com

Removed unused parameter.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201021044031.1752624-3-airlied@gmail.com

This moves the call to tt binding into the driver move,
and drops the driver callback.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201020010319.1692445-8-airlied@gmail.com

The drivers now do this in the move callback.

move_notify is still needed in the destroy path.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201020010319.1692445-7-airlied@gmail.com

This show the remaining bind callback, which my next series of
patches will aim to remove.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201020010319.1692445-6-airlied@gmail.com

The drivers now control this, so drop unbinding.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201020010319.1692445-5-airlied@gmail.com

This was adding size to start, but pfn and start are in pages,
so it should be using num_pages.

Not sure this fixes anything in the real world, just noticed it
during refactoring.
Signed-off-by: NDave Airlie <airlied@redhat.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Link: https://patchwork.freedesktop.org/patch/msgid/20201019222257.1684769-2-airlied@gmail.com

This factors out the code to setup non-system tt.

The same code was used twice in the move paths.
Reviewed-by: NBen Skeggs <bskeggs@redhat.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201019071314.1671485-2-airlied@gmail.com

Changing the caching on the fly never really worked
flawlessly.

So stop this completely and just let drivers specific the
desired caching in the tt or bus object.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NMichael J. Ruhl <michael.j.ruhl@intel.com>
Link: https://patchwork.freedesktop.org/patch/394256/

It is the sole user of this.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/393498/

We can always access the global state.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/393499/

Make it more clear what the resource manager function
does and nuke the wrapper function.

v2: nuke the wrapper
v3: fix typo in radeon, rebased
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch> (v2)
Link: https://patchwork.freedesktop.org/patch/393914/

All drivers should have a move callback now so make it compulsory.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NBen Skeggs <bskeggs@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201006000644.1005758-6-airlied@gmail.com

This just consolidates the code making the flow easier to understand
and also helps when moving move to the driver side.

Reviewed-by: Christian König <christian.koenig@amd.com>.
Signed-off-by: NDave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200924051845.397177-3-airlied@gmail.com

Not used any more.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NDave Airlie <airlied@redhat.com>
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/391604/?series=81973&rev=1

Not used any more.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NDave Airlie <airlied@redhat.com>
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/391616/?series=81973&rev=1