Replace the goto loop with a simple for each loop, and only run the
delayed destroy cleanup if we can reserve the buffer first.

No race occurs, since lru lock is never dropped any more. An empty list
and a list full of unreservable buffers both cause -EBUSY to be returned,
which is identical to the previous situation, because previously buffers
on the lru list were always guaranteed to be reservable.

This should work since currently ttm guarantees items on the lru are
always reservable, and reserving items blockingly with some bo held
are enough to cause you to run into a deadlock.

Currently this is not a concern since removal off the lru list and
reservations are always done with atomically, but when this guarantee
no longer holds, we have to handle this situation or end up with
possible deadlocks.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Replace the while loop with a simple for each loop, and only run the
delayed destroy cleanup if we can reserve the buffer first.

No race occurs, since lru lock is never dropped any more. An empty list
and a list full of unreservable buffers both cause -EBUSY to be returned,
which is identical to the previous situation.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

By removing the unlocking of lru and retaking it immediately, a race is
removed where the bo is taken off the swap list or the lru list between
the unlock and relock. As such the cleanup_refs code can be simplified,
it will attempt to call ttm_bo_wait non-blockingly, and if it fails
it will drop the locks and perform a blocking wait, or return an error
if no_wait_gpu was set.

The need for looping is also eliminated, since swapout and evict_mem_first
will always follow the destruction path, no new fence is allowed
to be attached. As far as I can see this may already have been the case,
but the unlocking / relocking required a complicated loop to deal with
re-reservation.

Changes since v1:
 - Simplify no_wait_gpu case by folding it in with empty ddestroy.
 - Hold a reservation while calling ttm_bo_cleanup_memtype_use again.
Changes since v2:
 - Do not remove bo from lru list while waiting
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This requires changing the order in ttm_bo_cleanup_refs_or_queue to
take the reservation first, as there is otherwise no race free way to
take lru lock before fence_lock.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Removes the need for a write lock each time we call ttm_bo_unref().

v2: Remove an unused variable.
v3: Really remove the unused variable.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This is similar to other platforms that don't allow command submission
to buffers locked on the cpu.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Reservation locking currently always takes place under the LRU spinlock.
Hence, strictly there is no need for an atomic_cmpxchg call; we can use
atomic_read followed by atomic_write since nobody else will ever reserve
without the lru spinlock held.
At least on Intel this should remove a locked bus cycle on successful
reserve.

Note that thit commit may be obsoleted by the cross-device reservation work.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-By: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

vmwgfx was its only user and always sets it to the same..
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-By: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

It's unused.
Signed-off-by: NMarcin Slusarz <marcin.slusarz@gmail.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

All drivers set it to 0 and nothing uses it.
Signed-off-by: NMarcin Slusarz <marcin.slusarz@gmail.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

In theory, that function could release the lru lock between
checking for bo on ddestroy list and a successful reserve if the bo
was already reserved, and the function was called with waiting reserves
allowed.
However, all current reservers of a bo on the ddestroy list would
atomically take the bo off the list after a successful reserve so this
race should not have been hit, so no need to backport for stable.

This patch also fixes a case found by Maarten Lankhorst where
ttm_mem_evict_first called with no_wait_gpu would incorrectly
spin waiting for bo idle if trying to evict a busy buffer that
also sits on the ddestroy list.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

The ttm_mem_evict_first function could theoretically drop the
lru lock without retrying if a reservation from off the LRU list
ended up waiting.
However, since currently there are no users that could cause a wait
in that situation so this is not suitable for stable
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Convert #include "..." to #include <path/...> in drivers/gpu/.
Signed-off-by: NDavid Howells <dhowells@redhat.com>
Acked-by: NDave Airlie <airlied@redhat.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NThomas Gleixner <tglx@linutronix.de>
Acked-by: NPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: NDave Jones <davej@redhat.com>

A regression was introduced in the 3.3 rc series, commit
"drm/ttm: simplify memory accounting for ttm user v2",
causing the metadata of buffer objects created using the ttm_bo_create()
function to be accounted twice.
That causes massive leaks with the vmwgfx driver running for example
SpecViewperf Catia-03 test 2, eventually killing the app.

Furthermore, the same commit introduces a regression where
metadata accounting is leaked if a buffer object is
initialized with an illegal size. This is also fixed with this commit.

v2: Fixed an error path and removed an unused variable.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: NDave Airlie <airlied@redhat.com>

This imbalance may cause hangs when TTM is trying to swap out a buffer
that is already on the delayed delete list.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NJakob Bornecrantz <jakob@vmware.com>
Cc: stable@vger.kernel.org
Signed-off-by: NDave Airlie <airlied@redhat.com>

This adds the ability for ttm common code to take an SG table
and use it as the backing for a slave TTM object.

The drivers can then populate their GTT tables using the SG object.

v2: make sure to setup VM for sg bos as well.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Use the more current logging style.

Add pr_fmt and remove the TTM_PFX uses.
Coalesce formats and align arguments.
Signed-off-by: NJoe Perches <joe@perches.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Both changes in dc97b340 cause serious
regressions in the nouveau driver.

move_notify() was originally able to presume that bo->mem is the old node,
and new_mem is the new node.  The above commit moves the call to
move_notify() to after move() has been done, which means that now, sometimes,
new_mem isn't the new node at all, bo->mem is, and new_mem points at a
stale, possibly-just-been-killed-by-move node.

This is clearly not a good situation.  This patch reverts this change, and
replaces it with a cleanup in the move() failure path instead.

The second issue is that the call to move_notify() from cleanup_memtype_use()
causes the TTM ghost objects to get passed into the driver.  This is clearly
bad as the driver knows nothing about these "fake" TTM BOs, and ends up
accessing uninitialised memory.

I worked around this in nouveau's move_notify() hook by ensuring the BO
destructor was nouveau's.  I don't particularly like this solution, and
would rather TTM never pass the driver these objects.  However, I don't
clearly understand the reason why we're calling move_notify() here anyway
and am happy to work around the problem in nouveau instead of breaking the
behaviour expected by other drivers.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Cc: Jerome Glisse <j.glisse@gmail.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Previously we were calling back move_notify in error path when the
bo is returned to it's original position or when destroy the bo.
When destroying the bo set the new mem placement as NULL when calling
back in the driver.

Updating nouveau to deal with NULL placement properly.

v2: reserve the object before calling move_notify in bo destroy path
    at that point ttm should be the only piece of code interacting
    with the object so atomic_set is safe here.
v3: callback move notify only once the bo is in its new position
    call move notify want swaping out the buffer
v4:- don't call move_notify when swapin out bo, assume driver should
     do what is appropriate in swap notify
   - move move_notify call back to ttm_bo_cleanup_memtype_use for
     destroy path
Reviewed-by: NJerome Glisse <jglisse@redhat.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>

Provide helper function to compute the kernel memory size needed
for each buffer object. Move all the accounting inside ttm, simplifying
driver and avoiding code duplication accross them.

v2 fix accounting of ghost object, one would have thought that i
   would have run into the issue since a longtime but it seems
   ghost object are rare when you have plenty of vram ;)
Signed-off-by: NJerome Glisse <jglisse@redhat.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>

ttm_backend will only exist with a ttm_tt, and ttm_tt
will only be of interest when bound to a backend. Merge them
to avoid code and data duplication.

V2 Rebase on top of memory accounting overhaul
V3 Rebase on top of more memory accounting changes
V4 Rebase on top of no memory account changes (where/when is my
   delorean when i need it ?)
V5 make sure ttm is unbound before destroying, change commit
   message on suggestion from Tormod Volden
Signed-off-by: NJerome Glisse <jglisse@redhat.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>

This was never use in none of the driver, properly using userspace
page for bo would need more code (vma interaction mostly). Removing
this dead code in preparation of ttm_tt & backend merge.
Signed-off-by: NJerome Glisse <jglisse@redhat.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>

An unlikely race could case a bo to be returned reserved on an error path.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This reverts commit dfadbbdb.

Further upstream discussion between Marek and Thomas decided this wasn't
fully baked and needed further work, so revert it before it hits mainline.
Signed-off-by: NDave Airlie <airlied@redhat.com>

Used by the vmwgfx driver.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Fixes an information leak to userspace, we were handing out un-zeroed pages
for any newly created TTM_PL_TT buffer.
Reported-by: NMarcin Slusarz <marcin.slusarz@gmail.com>
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>
Tested-by: NMarcin Slusarz <marcin.slusarz@gmail.com>
Cc: stable@kernel.org
Signed-off-by: NDave Airlie <airlied@redhat.com>

Sometimes we want to know whether a buffer is busy and wait for it (bo_wait).
However, sometimes it would be more useful to be able to query whether
a buffer is busy and being either read or written, and wait until it's stopped
being either read or written. The point of this is to be able to avoid
unnecessary waiting, e.g. if a GPU has written something to a buffer and is now
reading that buffer, and a CPU wants to map that buffer for read, it needs to
only wait for the last write. If there were no write, there wouldn't be any
waiting needed.

This, or course, requires user space drivers to send read/write flags
with each relocation (like we have read/write domains in radeon, so we can
actually use those for something useful now).

Now how this patch works:

The read/write flags should passed to ttm_validate_buffer. TTM maintains
separate sync objects of the last read and write for each buffer, in addition
to the sync object of the last use of a buffer. ttm_bo_wait then operates
with one the sync objects.
Signed-off-by: NMarek Olšák <maraeo@gmail.com>
Reviewed-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This was true for new TTM_PL_SYSTEM and new TTM_PL_TT cases, but wasn't
the case on TTM_PL_SYSTEM<->TTM_PL_TT moves, which causes trouble on some
paths as nouveau's move_notify() hook requires that the dma addresses be
valid at this point.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

ttm_tt_destroy kfrees passed object, so we need to nullify
a reference to it.
Signed-off-by: NMarcin Slusarz <marcin.slusarz@gmail.com>
Cc: stable@kernel.org
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This allows us to move duplicated code in <asm/atomic.h>
(atomic_inc_not_zero() for now) to <linux/atomic.h>
Signed-off-by: NArun Sharma <asharma@fb.com>
Reviewed-by: NEric Dumazet <eric.dumazet@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: David Miller <davem@davemloft.net>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Acked-by: NMike Frysinger <vapier@gentoo.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Signed-off-by: NJan Engelhardt <jengelh@medozas.de>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Nouveau doesn't have enough information at ttm_backend_func.bind() time
to implement things like tiled GART, or to keep a buffer at a constant
address in the GPU virtual address space no matter where in physical
memory it's placed.

To resolve this, nouveau will handle binding of all buffers to the GPU
itself from the move_notify() hook.  This commit ensures it's called
for all buffer moves.

Talked to Dave about the impact on radeon, which uses move_notify, it
doesn't look like anything should break there.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>
Reviewed-by: NThomas Hellstrom <thomas@shipmail.org>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Make ttm_bo::ttm_bo_device_release call cancel_delayed_work_sync()
instead of calling cancel_delayed_work() followed by
flush_scheduled_work().

This is to prepare for the deprecation and removal of
flush_scheduled_work().
Signed-off-by: NTejun Heo <tj@kernel.org>
Cc:: Thomas Hellstrom <thellstrom@vmware.com>
Cc:: Dave Airlie <airlied@redhat.com>

This patch attempts to fix up shortcomings with the current calling
sequences.

1) There's a fastpath where no locking occurs and only io_mem_reserved is
   called to obtain needed info for mapping. The fastpath is set per
   memory type manager.
2) If the fastpath is disabled, io_mem_reserve and io_mem_free will be exactly
   balanced and not called recursively for the same struct ttm_mem_reg.
3) Optionally the driver can choose to enable a per memory type manager LRU
   eviction mechanism that, when io_mem_reserve returns -EAGAIN will attempt
   to kill user-space mappings of memory in that manager to free up needed
   resources
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NBen Skeggs <bskeggs@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Rather than having the driver supply the validation sequence, leave that
responsibility to TTM. This saves some confusion and a function argument.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Drastically reduce the number of spin lock / unlock operations by performing
unreserving and fencing under global locks.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NJerome Glisse <j.glisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

The bo lock used only to protect the bo sync object members, and since it
is a per bo lock, fencing a buffer list will see a lot of locks and unlocks.
Replace it with a per-device lock that protects the sync object members on
*all* bos. Reading and setting these members will always be very quick, so
the risc of heavy lock contention is microscopic. Note that waiting for
sync objects will always take place outside of this lock.

The bo device fence lock will eventually be replaced with a seqlock /
rcu mechanism so we can determine that a bo is idle under a
rcu / read seqlock.

However this change will allow us to batch fencing and unreserving of
buffers with a minimal amount of locking.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NJerome Glisse <j.glisse@gmail.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Add an aid for the driver to detect deadlocks on multi-bo reservations
Update documentation.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NJerome Glisse <j.glisse@gmail.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>