Instead of dropping everything, waiting for the bo to be unreserved
and trying over, a better strategy would be to do a blocking wait.

This can be mapped a lot better to a mutex_lock-like call.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NJerome Glisse <jglisse@redhat.com>

With the lru lock no longer required for protecting reservations we
can just do a ttm_bo_reserve_nolru on -EBUSY, and handle all errors
in a single path.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NJerome Glisse <jglisse@redhat.com>

There should no longer be assumptions that reserve will always succeed
with the lru lock held, so we can safely break the whole atomic
reserve/lru thing. As a bonus this fixes most lockdep annotations for
reservations.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NJerome Glisse <jglisse@redhat.com>

Noticed while reviewing the fence locking in the radeon pageflip
handler.

v2: Instead of grabbing the bdev->fence_lock in object_transfer just
move the single callsite of that function a few lines, so that it is
protected by the fence_lock. Suggested by Jerome Glisse.

v3: Fix typo in commit message.
Reviewed-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Fix regression introduced by 85b144f8
"drm/ttm: call ttm_bo_cleanup_refs with reservation and lru lock held, v3"

Slowpath ttm_bo_cleanup_refs_and_unlock accidentally tried to increase
refcount on &bo->sync_obj instead of bo->sync_obj.

The compiler didn't complain since sync_obj_ref takes a void pointer,
so it was still valid c.

This could result in lockups, memory corruptions, and warnings like
these when graphics card VRAM usage is high:

------------[ cut here ]------------
WARNING: at include/linux/kref.h:42 radeon_fence_ref+0x2c/0x40()
Hardware name: System Product Name
Pid: 157, comm: X Not tainted 3.7.0-rc7-00520-g85b144f8-dirty #174
Call Trace:
[<ffffffff81058c84>] ? warn_slowpath_common+0x74/0xb0
[<ffffffff8129273c>] ? radeon_fence_ref+0x2c/0x40
[<ffffffff8125e95c>] ? ttm_bo_cleanup_refs_and_unlock+0x18c/0x2d0
[<ffffffff8125f17c>] ? ttm_mem_evict_first+0x1dc/0x2a0
[<ffffffff81264452>] ? ttm_bo_man_get_node+0x62/0xb0
[<ffffffff8125f4ce>] ? ttm_bo_mem_space+0x28e/0x340
[<ffffffff8125fb0c>] ? ttm_bo_move_buffer+0xfc/0x170
[<ffffffff810de172>] ? kmem_cache_alloc+0xb2/0xc0
[<ffffffff8125fc15>] ? ttm_bo_validate+0x95/0x110
[<ffffffff8125ff7c>] ? ttm_bo_init+0x2ec/0x3b0
[<ffffffff8129419a>] ? radeon_bo_create+0x18a/0x200
[<ffffffff81293e80>] ? radeon_bo_clear_va+0x40/0x40
[<ffffffff812a5342>] ? radeon_gem_object_create+0x92/0x160
[<ffffffff812a575c>] ? radeon_gem_create_ioctl+0x6c/0x150
[<ffffffff812a529f>] ? radeon_gem_object_free+0x2f/0x40
[<ffffffff81246b60>] ? drm_ioctl+0x420/0x4f0
[<ffffffff812a56f0>] ? radeon_gem_pwrite_ioctl+0x20/0x20
[<ffffffff810f53a4>] ? do_vfs_ioctl+0x2e4/0x4e0
[<ffffffff810e5588>] ? vfs_read+0x118/0x160
[<ffffffff810f55ec>] ? sys_ioctl+0x4c/0xa0
[<ffffffff810e5851>] ? sys_read+0x51/0xa0
[<ffffffff814b0612>] ? system_call_fastpath+0x16/0x1b
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reported-by: NMarkus Trippelsdorf <markus@trippelsdorf.de>
Acked-by: NPaul Menzel <paulepanter@users.sourceforge.net>
Signed-off-by: NDave Airlie <airlied@redhat.com>

All items on the lru list are always reservable, so this is a stupid
thing to keep. Not only that, it is used in a way which would
guarantee deadlocks if it were ever to be set to block on reserve.

This is a lot of churn, but mostly because of the removal of the
argument which can be nested arbitrarily deeply in many places.

No change of code in this patch except removal of the no_wait_reserve
argument, the previous patch removed the use of no_wait_reserve.

v2:
 - Warn if -EBUSY is returned on reservation, all objects on the list
   should be reservable. Adjusted patch slightly due to conflicts.
v3:
 - Focus on no_wait_reserve removal only.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Replace the goto loop with a simple for each loop, and only run the
delayed destroy cleanup if we can reserve the buffer first.

No race occurs, since lru lock is never dropped any more. An empty list
and a list full of unreservable buffers both cause -EBUSY to be returned,
which is identical to the previous situation, because previously buffers
on the lru list were always guaranteed to be reservable.

This should work since currently ttm guarantees items on the lru are
always reservable, and reserving items blockingly with some bo held
are enough to cause you to run into a deadlock.

Currently this is not a concern since removal off the lru list and
reservations are always done with atomically, but when this guarantee
no longer holds, we have to handle this situation or end up with
possible deadlocks.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Replace the while loop with a simple for each loop, and only run the
delayed destroy cleanup if we can reserve the buffer first.

No race occurs, since lru lock is never dropped any more. An empty list
and a list full of unreservable buffers both cause -EBUSY to be returned,
which is identical to the previous situation.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

By removing the unlocking of lru and retaking it immediately, a race is
removed where the bo is taken off the swap list or the lru list between
the unlock and relock. As such the cleanup_refs code can be simplified,
it will attempt to call ttm_bo_wait non-blockingly, and if it fails
it will drop the locks and perform a blocking wait, or return an error
if no_wait_gpu was set.

The need for looping is also eliminated, since swapout and evict_mem_first
will always follow the destruction path, no new fence is allowed
to be attached. As far as I can see this may already have been the case,
but the unlocking / relocking required a complicated loop to deal with
re-reservation.

Changes since v1:
 - Simplify no_wait_gpu case by folding it in with empty ddestroy.
 - Hold a reservation while calling ttm_bo_cleanup_memtype_use again.
Changes since v2:
 - Do not remove bo from lru list while waiting
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This requires changing the order in ttm_bo_cleanup_refs_or_queue to
take the reservation first, as there is otherwise no race free way to
take lru lock before fence_lock.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Removes the need for a write lock each time we call ttm_bo_unref().

v2: Remove an unused variable.
v3: Really remove the unused variable.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Also move a kref_init() out of spinlocked region
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This is similar to other platforms that don't allow command submission
to buffers locked on the cpu.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Reservation locking currently always takes place under the LRU spinlock.
Hence, strictly there is no need for an atomic_cmpxchg call; we can use
atomic_read followed by atomic_write since nobody else will ever reserve
without the lru spinlock held.
At least on Intel this should remove a locked bus cycle on successful
reserve.

Note that thit commit may be obsoleted by the cross-device reservation work.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

The mostly used lookup+get put+potential_destroy path of TTM objects
is converted to use RCU locks. This will substantially decrease the amount
of locked bus cycles during normal operation.
Since we use kfree_rcu to free the objects, no rcu synchronization is needed
at module unload time.

v2: Don't touch include/linux/kref.h
v3: Adapt to kref_get_unless_zero return value change
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-By: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-By: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

vmwgfx was its only user and always sets it to the same..
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-By: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

It's unused.
Signed-off-by: NMarcin Slusarz <marcin.slusarz@gmail.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

It's unused.
Signed-off-by: NMarcin Slusarz <marcin.slusarz@gmail.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

All drivers set it to 0 and nothing uses it.
Signed-off-by: NMarcin Slusarz <marcin.slusarz@gmail.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

It is unnecessary to disable preemption explicitly while calling
copy_highpage().  Because copy_highpage() will do it again through
kmap_atomic/kunmap_atomic.
Signed-off-by: NAkinobu Mita <akinobu.mita@gmail.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

The TTM page can be allocated from high memory. In such case it is
wrong to use the page_address(page) as the virtual address for the high memory
page.

bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=50241Signed-off-by: NZhao Yakui <yakui.zhao@intel.com>
Cc: stable@vger.kernel.org
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

In theory, that function could release the lru lock between
checking for bo on ddestroy list and a successful reserve if the bo
was already reserved, and the function was called with waiting reserves
allowed.
However, all current reservers of a bo on the ddestroy list would
atomically take the bo off the list after a successful reserve so this
race should not have been hit, so no need to backport for stable.

This patch also fixes a case found by Maarten Lankhorst where
ttm_mem_evict_first called with no_wait_gpu would incorrectly
spin waiting for bo idle if trying to evict a busy buffer that
also sits on the ddestroy list.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

The ttm_mem_evict_first function could theoretically drop the
lru lock without retrying if a reservation from off the LRU list
ended up waiting.
However, since currently there are no users that could cause a wait
in that situation so this is not suitable for stable
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

A long time ago, in v2.4, VM_RESERVED kept swapout process off VMA,
currently it lost original meaning but still has some effects:

 | effect                 | alternative flags
-+------------------------+---------------------------------------------
1| account as reserved_vm | VM_IO
2| skip in core dump      | VM_IO, VM_DONTDUMP
3| do not merge or expand | VM_IO, VM_DONTEXPAND, VM_HUGETLB, VM_PFNMAP
4| do not mlock           | VM_IO, VM_DONTEXPAND, VM_HUGETLB, VM_PFNMAP

This patch removes reserved_vm counter from mm_struct.  Seems like nobody
cares about it, it does not exported into userspace directly, it only
reduces total_vm showed in proc.

Thus VM_RESERVED can be replaced with VM_IO or pair VM_DONTEXPAND | VM_DONTDUMP.

remap_pfn_range() and io_remap_pfn_range() set VM_IO|VM_DONTEXPAND|VM_DONTDUMP.
remap_vmalloc_range() set VM_DONTEXPAND | VM_DONTDUMP.

[akpm@linux-foundation.org: drivers/vfio/pci/vfio_pci.c fixup]
Signed-off-by: NKonstantin Khlebnikov <khlebnikov@openvz.org>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Carsten Otte <cotte@de.ibm.com>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Eric Paris <eparis@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Morris <james.l.morris@oracle.com>
Cc: Jason Baron <jbaron@redhat.com>
Cc: Kentaro Takeda <takedakn@nttdata.co.jp>
Cc: Matt Helsley <matthltc@us.ibm.com>
Cc: Nick Piggin <npiggin@kernel.dk>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Robert Richter <robert.richter@amd.com>
Cc: Suresh Siddha <suresh.b.siddha@intel.com>
Cc: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Cc: Venkatesh Pallipadi <venki@google.com>
Acked-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Convert #include "..." to #include <path/...> in drivers/gpu/.
Signed-off-by: NDavid Howells <dhowells@redhat.com>
Acked-by: NDave Airlie <airlied@redhat.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NThomas Gleixner <tglx@linutronix.de>
Acked-by: NPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: NDave Jones <davej@redhat.com>

Remove useless kfree() and clean up code related to the removal.

The semantic patch that finds this problem is as follows:
(http://coccinelle.lip6.fr/)

// <smpl>
@r exists@
position p1,p2;
expression x;
@@

if (x@p1 == NULL) { ... kfree@p2(x); ... return ...; }

@unchanged exists@
position r.p1,r.p2;
expression e <= r.x,x,e1;
iterator I;
statement S;
@@

if (x@p1 == NULL) { ... when != I(x,...) S
                        when != e = e1
                        when != e += e1
                        when != e -= e1
                        when != ++e
                        when != --e
                        when != e++
                        when != e--
                        when != &e
   kfree@p2(x); ... return ...; }

@ok depends on unchanged exists@
position any r.p1;
position r.p2;
expression x;
@@

... when != true x@p1 == NULL
kfree@p2(x);

@depends on !ok && unchanged@
position r.p2;
expression x;
@@

*kfree@p2(x);
// </smpl>
Signed-off-by: NPeter Senna Tschudin <peter.senna@gmail.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Use copy_highpage() to copy from one page to another.
Signed-off-by: NAkinobu Mita <akinobu.mita@gmail.com>
Cc: dri-devel@lists.freedesktop.org
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NHuacai Chen <chenhc@lemote.com>
Signed-off-by: NHongliang Tao <taohl@lemote.com>
Signed-off-by: NHua Yan <yanh@lemote.com>
Cc: dri-devel@lists.freedesktop.org
Signed-off-by: NDave Airlie <airlied@redhat.com>

A regression was introduced in the 3.3 rc series, commit
"drm/ttm: simplify memory accounting for ttm user v2",
causing the metadata of buffer objects created using the ttm_bo_create()
function to be accounted twice.
That causes massive leaks with the vmwgfx driver running for example
SpecViewperf Catia-03 test 2, eventually killing the app.

Furthermore, the same commit introduces a regression where
metadata accounting is leaked if a buffer object is
initialized with an illegal size. This is also fixed with this commit.

v2: Fixed an error path and removed an unused variable.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: NDave Airlie <airlied@redhat.com>

This imbalance may cause hangs when TTM is trying to swap out a buffer
that is already on the delayed delete list.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NJakob Bornecrantz <jakob@vmware.com>
Cc: stable@vger.kernel.org
Signed-off-by: NDave Airlie <airlied@redhat.com>

This adds the ability for ttm common code to take an SG table
and use it as the backing for a slave TTM object.

The drivers can then populate their GTT tables using the SG object.

v2: make sure to setup VM for sg bos as well.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NCong Wang <amwang@redhat.com>

Use the more current logging style.

Add pr_fmt and remove the TTM_PFX uses.
Coalesce formats and align arguments.
Signed-off-by: NJoe Perches <joe@perches.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Both changes in dc97b340 cause serious
regressions in the nouveau driver.

move_notify() was originally able to presume that bo->mem is the old node,
and new_mem is the new node.  The above commit moves the call to
move_notify() to after move() has been done, which means that now, sometimes,
new_mem isn't the new node at all, bo->mem is, and new_mem points at a
stale, possibly-just-been-killed-by-move node.

This is clearly not a good situation.  This patch reverts this change, and
replaces it with a cleanup in the move() failure path instead.

The second issue is that the call to move_notify() from cleanup_memtype_use()
causes the TTM ghost objects to get passed into the driver.  This is clearly
bad as the driver knows nothing about these "fake" TTM BOs, and ends up
accessing uninitialised memory.

I worked around this in nouveau's move_notify() hook by ensuring the BO
destructor was nouveau's.  I don't particularly like this solution, and
would rather TTM never pass the driver these objects.  However, I don't
clearly understand the reason why we're calling move_notify() here anyway
and am happy to work around the problem in nouveau instead of breaking the
behaviour expected by other drivers.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Cc: Jerome Glisse <j.glisse@gmail.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

. It was useful during development, but now on a production system
we can get this (if the user forgot to upload the firmware):

[drm] radeon: irq initialized.
[drm] GART: num cpu pages 131072, num gpu pages 131072
[drm] radeon: ib pool ready.
[drm] Loading SUMO Microcode
r600_cp: Failed to load firmware "radeon/SUMO_pfp.bin"
atl1c 0000:03:00.0: version 1.0.1.0-NAPI.213057] [drm:evergreen_startup] *ERROR* Failed to load firmware!
radeon 0000:00:01.0: disabling GPU acceleration
88] radeon 0000:00:01.0: ffff8801bb782400 unpin not necessary
------------[ cut here ]------------
WARNING: at /home/konrad/linux-linus/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c:956 ttm_dma_unpopulate+0x79/0x300 [ttm]()
Hardware name: System Product Name
Modules linked in: e1000e atl1c radeon(+) ahci libahci libata scsi_mod fbcon tileblit font ttm bitblit softcursor drm_kms_helper wmi xen_blkfront xen_netfront fb_sys_fops sysimgblt sysfillrect syscopyarea xenfs xen_privcmd
Pid: 1600, comm: modprobe Not tainted 3.2.0-06100-ge343a895 #1
Call Trace:
 [<ffffffff8108973a>] warn_slowpath_common+0x7a/0xb0
 [<ffffffff81089785>] warn_slowpath_null+0x15/0x20
 [<ffffffffa0060309>] ttm_dma_unpopulate+0x79/0x300 [ttm]
 [<ffffffffa01341c0>] radeon_ttm_tt_unpopulate+0x120/0x130 [radeon]
 [<ffffffffa0056e0c>] ttm_tt_destroy+0x2c/0x70 [ttm]
 [<ffffffffa0057a4e>] ttm_bo_cleanup_memtype_use+0x3e/0x80 [ttm]
 [<ffffffffa00595a1>] ttm_bo_release+0x251/0x280 [ttm]
 [<ffffffffa0059610>] ttm_bo_unref+0x40/0x60 [ttm]
 [<ffffffffa0134d02>] radeon_bo_unref+0x42/0x80 [radeon]
 [<ffffffffa0186dfb>] radeon_sa_bo_manager_fini+0x6b/0x80 [radeon]
 [<ffffffffa0146b8f>] radeon_ib_pool_fini+0x6f/0x90 [radeon]
 [<ffffffffa014be49>] r100_ib_fini+0x19/0x20 [radeon]
 [<ffffffffa017b47e>] evergreen_init+0x1ee/0x2d0 [radeon]

The big WARN() has nothing to do with the culprit - which is that
the firmware was not loaded. So lets remove the WARN() from the TTM DMA code.
Signed-off-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

The "if (!p && !p->dev)" condition isn't right because || was intended
instead of &&.  But actually, "p" is the list cursor and so it's always
non-NULL and we can just remove that bit.  We can remove the another
similar check as well.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Reviewed-by: NJerome Glisse <jglisse@redhat.com>
Acked-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>