Not allowing errors here is completely pointless and actually dangerous
cause trying to continue on an error can cause an endless loop.
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NRoger.He <Hongbo.He@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Ensure that the driver can listen to evictions even when they don't take the
path through ttm_bo_driver::move.

This is crucial for amdgpu, which relies on an eviction counter to skip
re-binding page tables when possible.
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

reservation_object_wait_timeout_rcu() should enable signaling even with a
zero timeout, but ttm_bo_wait() can also be called from atomic context and
then it is not a good idea to do this.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NGustavo Padovan <gustavo.padovan@collabora.co.uk>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NSumit Semwal <sumit.semwal@linaro.org>
  [sumits: fix checkpatch warnings]
Link: http://patchwork.freedesktop.org/patch/msgid/1478553376-18575-3-git-send-email-alexander.deucher@amd.com

This way the driver can decide if it is valuable to evict a BO or not.

The current implementation is added as default to all existing drivers.

v2: fix some typos found during internal testing
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

I plan to usurp the short name of struct fence for a core kernel struct,
and so I need to rename the specialised fence/timeline for DMA
operations to make room.

A consensus was reached in
https://lists.freedesktop.org/archives/dri-devel/2016-July/113083.html
that making clear this fence applies to DMA operations was a good thing.
Since then the patch has grown a bit as usage increases, so hopefully it
remains a good thing!

(v2...: rebase, rerun spatch)
v3: Compile on msm, spotted a manual fixup that I broke.
v4: Try again for msm, sorry Daniel

coccinelle script:
@@

@@
- struct fence
+ struct dma_fence
@@

@@
- struct fence_ops
+ struct dma_fence_ops
@@

@@
- struct fence_cb
+ struct dma_fence_cb
@@

@@
- struct fence_array
+ struct dma_fence_array
@@

@@
- enum fence_flag_bits
+ enum dma_fence_flag_bits
@@

@@
(
- fence_init
+ dma_fence_init
|
- fence_release
+ dma_fence_release
|
- fence_free
+ dma_fence_free
|
- fence_get
+ dma_fence_get
|
- fence_get_rcu
+ dma_fence_get_rcu
|
- fence_put
+ dma_fence_put
|
- fence_signal
+ dma_fence_signal
|
- fence_signal_locked
+ dma_fence_signal_locked
|
- fence_default_wait
+ dma_fence_default_wait
|
- fence_add_callback
+ dma_fence_add_callback
|
- fence_remove_callback
+ dma_fence_remove_callback
|
- fence_enable_sw_signaling
+ dma_fence_enable_sw_signaling
|
- fence_is_signaled_locked
+ dma_fence_is_signaled_locked
|
- fence_is_signaled
+ dma_fence_is_signaled
|
- fence_is_later
+ dma_fence_is_later
|
- fence_later
+ dma_fence_later
|
- fence_wait_timeout
+ dma_fence_wait_timeout
|
- fence_wait_any_timeout
+ dma_fence_wait_any_timeout
|
- fence_wait
+ dma_fence_wait
|
- fence_context_alloc
+ dma_fence_context_alloc
|
- fence_array_create
+ dma_fence_array_create
|
- to_fence_array
+ to_dma_fence_array
|
- fence_is_array
+ dma_fence_is_array
|
- trace_fence_emit
+ trace_dma_fence_emit
|
- FENCE_TRACE
+ DMA_FENCE_TRACE
|
- FENCE_WARN
+ DMA_FENCE_WARN
|
- FENCE_ERR
+ DMA_FENCE_ERR
)
 (
 ...
 )
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NGustavo Padovan <gustavo.padovan@collabora.co.uk>
Acked-by: NSumit Semwal <sumit.semwal@linaro.org>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: http://patchwork.freedesktop.org/patch/msgid/20161025120045.28839-1-chris@chris-wilson.co.uk

A bit pointless to search for the first bit set manually.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Fixes hangs under memory pressure, e.g. running the piglit test
tex3d-maxsize concurrently with other tests.

Fixes: 17d33bc9 ("drm/ttm: drop waiting for idle in ttm_bo_evict.")
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Fixes hangs under memory pressure, e.g. running the piglit test
tex3d-maxsize concurrently with other tests.

Fixes: 17d33bc9 ("drm/ttm: drop waiting for idle in ttm_bo_evict.")
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The ttm_tt_destroy() function tests whether its argument is NULL
and then returns immediately. Thus the test around the call is not needed.

This issue was detected by using the Coccinelle software.
Signed-off-by: NMarkus Elfring <elfring@users.sourceforge.net>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: http://patchwork.freedesktop.org/patch/msgid/54338f58-830c-a8b4-4554-5d4459bcd321@users.sourceforge.net

Now that we can pipeline evictions we need to wait for
them to finish when we cleanup a memory domain.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Free up the memory immediately, remember the last eviction for each domain and
make new allocations depend on the last eviction to be completed.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

As far as I can see no need for a custom implementation any more.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Instead of using the flag just remember the fence of the last move operation.

This avoids waiting for command submissions pipelined after the move, but
before accessing the BO with the CPU again.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Final part to avoid pre move waits.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

That is unnecessary now.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

That is unnecessary now.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

When we want to pipeline accelerated moves we need to wait in the fallback path.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The function is a no-op with a NULL pointer.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

ttm_tt_destroy should be the only one unbinding the object.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

There are cases where it is desired to see if a proposed placement
is compatible with a buffer object before calling ttm_bo_validate().
Signed-off-by: NSinclair Yeh <syeh@vmware.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Cc: <stable@vger.kernel.org>
---
This is the first of a 3-patch series to fix a black screen
issue observed on Ubuntu 16.04 server.

This allows fine grained control for the driver where to add a BO into the LRU.

v2: fix typo in comment
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Useful for driver specific LRU handling.

v2: fix typo in comment
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Not used any more.
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Not used any more.
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Not used any more.
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

When we use an extern reservation object that otherwise waits for every
fence registered with it.
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

TTM BO accounting is out of sync with how memory is really allocated
in ttm[_dma]_tt_alloc_page_directory. This resulted in excessive
estimated overhead with many small allocations.

ttm_dma_tt_alloc_page_directory makes a single allocation for three
arrays: pages, DMA and CPU addresses. It uses drm_calloc_large, which
uses kmalloc internally for allocations smaller than PAGE_SIZE.
ttm_round_pot should be a good approximation of its memory usage both
above and below PAGE_SIZE.
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NMonk Liu <monk.liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Fixes the following scenario:

1. Page table bo allocated in vram and linked to man->lru.
   tbo->list_kref.refcount=2
2. Page table bo is swapped out and removed from man->lru.
   tbo->list_kref.refcount=1
3. Command submission from userspace.  Page table bo is moved
   to vram.  ttm_bo_move_to_lru_tail() link it to man->lru and
   don't increase the kref count.
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NFlora Cui <Flora.Cui@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

This allows the drivers to move a BO to the end of the LRU
without removing and adding it again.

v2: Make it more robust, handle pinned and swapable BOs as well.
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

It doesn't make any sense to try to swap out imported BOs.
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

If we import a BO with an external reservation object we don't
reserve/unreserve it. So we never add it to the LRU causing a possible
denial of service.

v2: fix typo in commit message
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

In the event that TTM doesn't find a compatible memory type for the
driver's first placement choice (placement without eviction), TTM
returns -EINVAL without trying the driver's second choice.
This causes problems on vmwgfx when VRAM is disabled before first modeset
and during VT switches when fbdev is not enabled.

Fix this by also trying the driver's second choice before returning
-EINVAL.

v2: Also check that man->use_type is true for the driver's second choice.
Fixes a bug where disallowed memory types could be used.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NBrian Paul <brianp@vmware.com>
Reviewed-by: NSinclair Yeh <syeh@vmware.com>

We need to store device offsets in 64 bit as the device
address space may be larger than the CPU's.

Fixes GPU init failures on radeons with 4GB or more of
vram on 32 bit kernels.  We put vram at the start of the
GPU's address space so the gart aperture starts at 4 GB
causing all GPU addresses in the gart aperture to get
truncated.

bug:
https://bugs.freedesktop.org/show_bug.cgi?id=89072

[airlied: fix warning on nouveau build]
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: thellstrom@vmware.com
Acked-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

The radeon driver uses placement range restrictions for several reasons,
in particular to make sure BOs in VRAM can be accessed by the CPU, e.g.
during a page fault.

Without this change, TTM could evict other BOs while trying to satisfy
the requested placement, even if the evicted BOs were outside of the
requested placement range. Doing so didn't free up any space in the
requested placement range, so the (potentially high) eviction cost was
incurred for no benefit.

Nominating for stable because radeon driver changes in 3.17 made this
much more noticeable than before.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=84662
Cc: stable@vger.kernel.org
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This allows importing reservation objects from dma-bufs.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>

Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>

Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>