When ttm_bo_init() fails, the reservation mutex should be unlocked.

In debug build, the kernel reported "possible recursive locking
detected" in this codepath. For debugging purposes, I also added
a "WARN_ON(ww_mutex_is_locked())" when ttm_bo_init() fails and the
mutex was locked as expected.

This should fix (random) GPU hangs. The easy way to reproduce the
issue is to change the "Super Sampling" option from 1.0 to 2.0 in
Hitman. It will create a huge buffer, evict a bunch of buffers
(around ~5k) and deadlock.

This regression has been introduced pretty recently.

v2: only release the mutex if resv is NULL

Fixes: 12a852219583 ("drm/amdgpu: improve AMDGPU_GEM_CREATE_VRAM_CLEARED handling (v2)")
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NSamuel Pitoiset <samuel.pitoiset@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

My randconfig tests on linux-next showed a newly introduced warning:

drivers/gpu/drm/amd/amdgpu/amdgpu_object.c: In function 'amdgpu_bo_create_restricted':
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c:377:2: error: #warning Please enable CONFIG_MTRR and CONFIG_X86_PAT for better performance thanks to write-combining [-Werror=cpp]

Generally speaking, warnings about bad kernel configuration are not particularly
helpful. We could enforce the selection of X86_PAT through Kconfig, so the driver
cannot even be used unless it is enabled, or we could just rely on the runtime
warning that is also there.

In this version, I'm making the warning conditional on CONFIG_COMPILE_TEST, which
shuts it up for me, but not people that may actually want to run the kernel
as a compromize.

Fixes: a2e2f299 ("drm/amdgpu: Bring bo creation in line with radeon driver (v2)")
Reviewed-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add the bo creation changes that have been done to the radeon driver in
recent times, e.g. disable GTT WC on 32 bit because it is broken there,
and also disable it generally (and print a warning message) when
CONFIG_X86_PAT is not set.

v2: agd: fix warning in defined(CONFIG_X86) && !defined(CONFIG_X86_PAT)
    case
Reviewed-by: NMichel Dänzer <michel.daenzer@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NNils Holland <nholland@tisys.org>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Give kernel allocations a higher priority cause it is often
more work to swap them back in.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NRoger.He <Hongbo.He@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Keeping groups of BOs on the LRU is to time consuming on command submission.

Instead use the newly added BO priority to give a certain eviction order.

v2: agd: trivial warning fix
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NRoger.He <Hongbo.He@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Unlock the resv lock only if we were the ones to lock it in the first
place.
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

ttm_bo_init checks that the reservation object is locked. This is
the caller's responsibility when resv != NULL. Otherwise, the inline
reservation object of the newly allocated buffer is used and must
explicitly be locked.

Uninterruptible w/w locks without an acquire context are always
successful.

v2: use ww_mutex_lock
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: Edward O'Callaghan <funfunctor@folklore1984.net> (v1)
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This catches evictions of shadow page tables from the GART. Since shadow
page tables are always stored in system memory, amdgpu_bo_move is never
called for them.

This fixes a crash during command submission that occurs when only a shadow
page table and no other BOs were evicted since the last submission.

Fixes: 1baa439f ("drm/amdgpu: allocate shadow for pd/pt bo V2")
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Ensure that the driver can listen to evictions even when they don't take the
path through ttm_bo_driver::move.

This is crucial for amdgpu, which relies on an eviction counter to skip
re-binding page tables when possible.
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Drop this whole reserve/unreserve dance, just lock the
reservation object manually when creating the BO.

v2: rebase on dma_fence renaming
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Manually checking why a function could fail is not a good idea
if you can just check the functions return code.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This only has the effect of scanning the invisible range twice
since the topdown flag is given anyway.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJimQu <Jim.Qu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This fixes a regression in all these drivers since the cache
mode tracking was fixed for mixed mappings. It uses the new
arch API to add the VRAM range to the PAT mapping tracking
tables.

Fixes: 87744ab3 (mm: fix cache mode tracking in vm_insert_mixed())
Reviewed-by: Christian König <christian.koenig@amd.com>.
Signed-off-by: NDave Airlie <airlied@redhat.com>

It's completely pointless to have two pointers to the
device in the same structure.

v2: rename function to amdgpu_ttm_adev, fix typos
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Split VRAM won't have a valid offset, so just set an explicit limit
when the flag is given to trigger reallocation if necessary.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Tested-by: NMike Lothian <mike@fireburn.co.uk>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add a flag noting that a BO must be created using linear VRAM
and set this flag on all in kernel users where appropriate.

Hopefully I haven't missed anything.

v2: add it in a few more places, fix CPU mapping.
v3: rename to VRAM_CONTIGUOUS, fix typo in CS code.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Tested-by: NMike Lothian <mike@fireburn.co.uk>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

I plan to usurp the short name of struct fence for a core kernel struct,
and so I need to rename the specialised fence/timeline for DMA
operations to make room.

A consensus was reached in
https://lists.freedesktop.org/archives/dri-devel/2016-July/113083.html
that making clear this fence applies to DMA operations was a good thing.
Since then the patch has grown a bit as usage increases, so hopefully it
remains a good thing!

(v2...: rebase, rerun spatch)
v3: Compile on msm, spotted a manual fixup that I broke.
v4: Try again for msm, sorry Daniel

coccinelle script:
@@

@@
- struct fence
+ struct dma_fence
@@

@@
- struct fence_ops
+ struct dma_fence_ops
@@

@@
- struct fence_cb
+ struct dma_fence_cb
@@

@@
- struct fence_array
+ struct dma_fence_array
@@

@@
- enum fence_flag_bits
+ enum dma_fence_flag_bits
@@

@@
(
- fence_init
+ dma_fence_init
|
- fence_release
+ dma_fence_release
|
- fence_free
+ dma_fence_free
|
- fence_get
+ dma_fence_get
|
- fence_get_rcu
+ dma_fence_get_rcu
|
- fence_put
+ dma_fence_put
|
- fence_signal
+ dma_fence_signal
|
- fence_signal_locked
+ dma_fence_signal_locked
|
- fence_default_wait
+ dma_fence_default_wait
|
- fence_add_callback
+ dma_fence_add_callback
|
- fence_remove_callback
+ dma_fence_remove_callback
|
- fence_enable_sw_signaling
+ dma_fence_enable_sw_signaling
|
- fence_is_signaled_locked
+ dma_fence_is_signaled_locked
|
- fence_is_signaled
+ dma_fence_is_signaled
|
- fence_is_later
+ dma_fence_is_later
|
- fence_later
+ dma_fence_later
|
- fence_wait_timeout
+ dma_fence_wait_timeout
|
- fence_wait_any_timeout
+ dma_fence_wait_any_timeout
|
- fence_wait
+ dma_fence_wait
|
- fence_context_alloc
+ dma_fence_context_alloc
|
- fence_array_create
+ dma_fence_array_create
|
- to_fence_array
+ to_dma_fence_array
|
- fence_is_array
+ dma_fence_is_array
|
- trace_fence_emit
+ trace_dma_fence_emit
|
- FENCE_TRACE
+ DMA_FENCE_TRACE
|
- FENCE_WARN
+ DMA_FENCE_WARN
|
- FENCE_ERR
+ DMA_FENCE_ERR
)
 (
 ...
 )
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NGustavo Padovan <gustavo.padovan@collabora.co.uk>
Acked-by: NSumit Semwal <sumit.semwal@linaro.org>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Link: http://patchwork.freedesktop.org/patch/msgid/20161025120045.28839-1-chris@chris-wilson.co.uk

Just to cleanup some radeon leftovers.

sed -i "s/rbo/abo/g" drivers/gpu/drm/amd/amdgpu/*.c
sed -i "s/rbo/abo/g" drivers/gpu/drm/amd/amdgpu/*.h

v2: rebased
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Not used in a while.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Only allocate address space when we really need it.

v2: fix a typo, add correct function description,
    stop leaking the node in the error case.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We get 2 warnings when building kernel with W=1:

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c:985:5: warning: no previous prototype for 'amdgpu_ttm_init' [-Wmissing-prototypes]
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c:1092:6: warning: no previous prototype for 'amdgpu_ttm_fini' [-Wmissing-prototypes]

In fact, both functions are declared in
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c, but should be declared in
a header file, thus can be recognized in other file.

So this patch moves the declarations into
drivers/gpu/drm/amd/amdgpu/amdgpu.h.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NBaoyou Xie <baoyou.xie@linaro.org>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

An invalid offset to be used by custom VRAM/GGT manager when we
don't have an offset for an BO.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NFlora Cui <Flora.Cui@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We don't really need the GTT table any more most of the time. So bind it
only on demand.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Acked-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Free the BO allocated by amdgpu_bo_create_kernel()
Signed-off-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

AMDGPU_GEM_CREATE_NO_CPU_ACCESS and AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED are
obviously mutual exclusive. So stop adding a dummy entry without effect when
both are specified.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Make it more clear what this function does. No intendet functional change.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

V2:
1. use mutex instead of spinlock for shadow list, since its process could
sleep.
2. move list_del to bo destroy phase.
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Use shadow flag to judge which direction to sync.
V2:
Don't need bo pin, so remove it.

V3:
1. Split to two functions, one is backup_to_shadow, another is
restore_from_shadow.
2. Clean up previous shadow direction difinitions.
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

V2:
add checking if need backup in amdgpu_bo_create.
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NFlora Cui <Flora.Cui@amd.com>
Reviewed-by: NKen Wang <Qingqing.Wang@amd.com>
Reviewed-by: NMichel Dänzer <michel.daenzer@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NFlora Cui <Flora.Cui@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

abort if the bo is pined to other domain already
Signed-off-by: NFlora Cui <Flora.Cui@amd.com>
Reviewed-by: NMichel Dänzer <michel.daenzer@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Just a few 80 chars problems.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

shadow bo is the shadow of a bo, which is always in GTT,
which can be used to backup the original bo.
V2:
reference shadow parent, shadow bo will be freed by who allocted him.
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

V2: fix the return value for fill failure and validate bo before
filling data
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NFlora Cui <Flora.Cui@amd.com>
Reviewed-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Warn when we try to get the address and the BO isn't locked or reserved.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We often allocate, pin and map things at the same time in the kernel.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

 - adding amdgpu_cs_bo_status to track total size and
   total entry count of bo for each submission.
 - adding amdgpu_ttm_bo_move to track the bo eviction
   including the size of bo and the location before/after the move
Signed-off-by: NDavid Mao <David.Mao@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Without this there was a double free of the metadata,
which ended up freeing the fd table for me here, and taking
out the machine more often than not.

I reproduced with X.org + modesetting DDX + latest llvm/mesa,
also required using dri3.

Cc: stable@vger.kernel.org
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>