As far as I can see no need for a custom implementation any more.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Instead of using the flag just remember the fence of the last move operation.

This avoids waiting for command submissions pipelined after the move, but
before accessing the BO with the CPU again.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Final part to avoid pre move waits.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

That is unnecessary now.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

That is unnecessary now.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

When we want to pipeline accelerated moves we need to wait in the fallback path.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The function is a no-op with a NULL pointer.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

ttm_tt_destroy should be the only one unbinding the object.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This allows fine grained control for the driver where to add a BO into the LRU.

v2: fix typo in comment
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Useful for driver specific LRU handling.

v2: fix typo in comment
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Not used any more.
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Not used any more.
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Not used any more.
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

When we use an extern reservation object that otherwise waits for every
fence registered with it.
Reviewed-by: NSinclair Yeh <syeh@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

TTM BO accounting is out of sync with how memory is really allocated
in ttm[_dma]_tt_alloc_page_directory. This resulted in excessive
estimated overhead with many small allocations.

ttm_dma_tt_alloc_page_directory makes a single allocation for three
arrays: pages, DMA and CPU addresses. It uses drm_calloc_large, which
uses kmalloc internally for allocations smaller than PAGE_SIZE.
ttm_round_pot should be a good approximation of its memory usage both
above and below PAGE_SIZE.
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NMonk Liu <monk.liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Fixes the following scenario:

1. Page table bo allocated in vram and linked to man->lru.
   tbo->list_kref.refcount=2
2. Page table bo is swapped out and removed from man->lru.
   tbo->list_kref.refcount=1
3. Command submission from userspace.  Page table bo is moved
   to vram.  ttm_bo_move_to_lru_tail() link it to man->lru and
   don't increase the kref count.
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NFlora Cui <Flora.Cui@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

This allows the drivers to move a BO to the end of the LRU
without removing and adding it again.

v2: Make it more robust, handle pinned and swapable BOs as well.
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

It doesn't make any sense to try to swap out imported BOs.
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

If we import a BO with an external reservation object we don't
reserve/unreserve it. So we never add it to the LRU causing a possible
denial of service.

v2: fix typo in commit message
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

In the event that TTM doesn't find a compatible memory type for the
driver's first placement choice (placement without eviction), TTM
returns -EINVAL without trying the driver's second choice.
This causes problems on vmwgfx when VRAM is disabled before first modeset
and during VT switches when fbdev is not enabled.

Fix this by also trying the driver's second choice before returning
-EINVAL.

v2: Also check that man->use_type is true for the driver's second choice.
Fixes a bug where disallowed memory types could be used.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NBrian Paul <brianp@vmware.com>
Reviewed-by: NSinclair Yeh <syeh@vmware.com>

We need to store device offsets in 64 bit as the device
address space may be larger than the CPU's.

Fixes GPU init failures on radeons with 4GB or more of
vram on 32 bit kernels.  We put vram at the start of the
GPU's address space so the gart aperture starts at 4 GB
causing all GPU addresses in the gart aperture to get
truncated.

bug:
https://bugs.freedesktop.org/show_bug.cgi?id=89072

[airlied: fix warning on nouveau build]
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: thellstrom@vmware.com
Acked-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

The radeon driver uses placement range restrictions for several reasons,
in particular to make sure BOs in VRAM can be accessed by the CPU, e.g.
during a page fault.

Without this change, TTM could evict other BOs while trying to satisfy
the requested placement, even if the evicted BOs were outside of the
requested placement range. Doing so didn't free up any space in the
requested placement range, so the (potentially high) eviction cost was
incurred for no benefit.

Nominating for stable because radeon driver changes in 3.17 made this
much more noticeable than before.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=84662
Cc: stable@vger.kernel.org
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This allows importing reservation objects from dma-bufs.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>

Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>

Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>

No users are left, kill it off! :D
Conversion to the reservation api is next on the list, after
that the functionality can be restored with rcu.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>

This is the last remaining function that doesn't use the reservation
lock completely to fence off access to a buffer.
Signed-off-by: NMaarten Lankhorst <maarten.lankhorst@canonical.com>

This allows us to more fine grained specify where to place the buffer object.

v2: rebased on drm-next, add bochs changes as well
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

Fix a sparse warning: ttm_bo_reserve()'s last argument is a
pointer to a struct, so use NULL as nullpointer.
Signed-off-by: NMartin Kepplinger <martink@posteo.de>
Reviewed-by: NJingoo Han <jg1.han@samsung.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

bo->mem.placement is not initialized when ttm_bo_man_get_node is called,
so the flag had no effect at all.

v2: change nouveau and vmwgfx as well
Signed-off-by: NChristian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NBrian Paul <brianp@vmware.com>

With dev->anon_inode we have a global address_space ready for operation
right from the beginning. Therefore, there is no need to do a delayed
setup with TTM. Instead, set dev_mapping during initialization in
ttm_bo_device_init() and remove any "if (dev_mapping)" conditions.

Cc: Dave Airlie <airlied@redhat.com>
Cc: Ben Skeggs <bskeggs@redhat.com>
Cc: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Cc: Alex Deucher <alexdeucher@gmail.com>
Cc: Thomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDavid Herrmann <dh.herrmann@gmail.com>

A few of the simpler TTM drivers (cirrus, ast, mgag200) do not implement
this function.  Yet can end up somehow with an evicted bo:

  BUG: unable to handle kernel NULL pointer dereference at           (null)
  IP: [<          (null)>]           (null)
  PGD 16e761067 PUD 16e6cf067 PMD 0
  Oops: 0010 [#1] SMP
  Modules linked in: bnep bluetooth rfkill fuse ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables sg btrfs zlib_deflate raid6_pq xor dm_queue_length iTCO_wdt iTCO_vendor_support coretemp kvm dcdbas dm_service_time microcode serio_raw pcspkr lpc_ich mfd_core i7core_edac edac_core ses enclosure ipmi_si ipmi_msghandler shpchp acpi_power_meter mperf nfsd auth_rpcgss nfs_acl lockd uinput sunrpc dm_multipath xfs libcrc32c ata_generic pata_acpi sr_mod cdrom
   sd_mod usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit lpfc drm_kms_helper ttm crc32c_intel ata_piix bfa drm ixgbe libata i2c_core mdio crc_t10dif ptp crct10dif_common pps_core scsi_transport_fc dca scsi_tgt megaraid_sas bnx2 dm_mirror dm_region_hash dm_log dm_mod
  CPU: 16 PID: 2572 Comm: X Not tainted 3.10.0-86.el7.x86_64 #1
  Hardware name: Dell Inc. PowerEdge R810/0H235N, BIOS 0.3.0 11/14/2009
  task: ffff8801799dabc0 ti: ffff88016c884000 task.ti: ffff88016c884000
  RIP: 0010:[<0000000000000000>]  [<          (null)>]           (null)
  RSP: 0018:ffff88016c885ad8  EFLAGS: 00010202
  RAX: ffffffffa04e94c0 RBX: ffff880178937a20 RCX: 0000000000000000
  RDX: 0000000000000000 RSI: 0000000000240004 RDI: ffff880178937a00
  RBP: ffff88016c885b60 R08: 00000000000171a0 R09: ffff88007cf171a0
  R10: ffffea0005842540 R11: ffffffff810487b9 R12: ffff880178937b30
  R13: ffff880178937a00 R14: ffff88016c885b78 R15: ffff880179929400
  FS:  00007f81ba2ef980(0000) GS:ffff88007cf00000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 0000000000000000 CR3: 000000016e763000 CR4: 00000000000007e0
  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  Stack:
   ffffffffa0306fae ffff8801799295c0 0000000000260004 0000000000000001
   ffff88016c885b60 ffffffffa0307669 00ff88007cf17738 ffff88017cf17700
   ffff880178937a00 ffff880100000000 ffff880100000000 0000000079929400
  Call Trace:
   [<ffffffffa0306fae>] ? ttm_bo_handle_move_mem+0x54e/0x5b0 [ttm]
   [<ffffffffa0307669>] ? ttm_bo_mem_space+0x169/0x340 [ttm]
   [<ffffffffa0307bd7>] ttm_bo_move_buffer+0x117/0x130 [ttm]
   [<ffffffff81130001>] ? perf_event_init_context+0x141/0x220
   [<ffffffffa0307cb1>] ttm_bo_validate+0xc1/0x130 [ttm]
   [<ffffffffa04e7377>] mgag200_bo_pin+0x87/0xc0 [mgag200]
   [<ffffffffa04e56c4>] mga_crtc_cursor_set+0x474/0xbb0 [mgag200]
   [<ffffffff811971d2>] ? __mem_cgroup_commit_charge+0x152/0x3b0
   [<ffffffff815c4182>] ? mutex_lock+0x12/0x2f
   [<ffffffffa0201433>] drm_mode_cursor_common+0x123/0x170 [drm]
   [<ffffffffa0205231>] drm_mode_cursor_ioctl+0x41/0x50 [drm]
   [<ffffffffa01f5ca2>] drm_ioctl+0x502/0x630 [drm]
   [<ffffffff815cbab4>] ? __do_page_fault+0x1f4/0x510
   [<ffffffff8101cb68>] ? __restore_xstate_sig+0x218/0x4f0
   [<ffffffff811b4445>] do_vfs_ioctl+0x2e5/0x4d0
   [<ffffffff8124488e>] ? file_has_perm+0x8e/0xa0
   [<ffffffff811b46b1>] SyS_ioctl+0x81/0xa0
   [<ffffffff815d05d9>] system_call_fastpath+0x16/0x1b
  Code:  Bad RIP value.
  RIP  [<          (null)>]           (null)
   RSP <ffff88016c885ad8>
  CR2: 0000000000000000
Signed-off-by: NRob Clark <rclark@redhat.com>
Reviewed-by: NJérôme Glisse <jglisse@redhat.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Cc: stable@vger.kernel.org

Mark function as static because it is not used outside file
drm/ttm/ttm_bo.c.

This eliminates the following warning in drm/ttm/ttm_bo.c:
drivers/gpu/drm/ttm/ttm_bo.c:960:5: warning: no previous prototype for ‘ttm_bo_move_buffer’ [-Wmissing-prototypes]
Signed-off-by: NRashika Kheria <rashika.kheria@gmail.com>
Reviewed-by: NJosh Triplett <josh@joshtriplett.org>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>

Addresses
"[BUG] completely bonkers use of set_need_resched + VM_FAULT_NOPAGE".

In the first occurence it was used to try to be nice while releasing the
mmap_sem and retrying the fault to work around a locking inversion.
The second occurence was never used.

There has been some discussion whether we should change the locking order to
mmap_sem -> bo_reserve. This patch doesn't address that issue, and leaves
that locking order undefined. The solution that we release the mmap_sem if
tryreserve fails and wait for the buffer to become unreserved is something
we want in any case, and follows how the core vm system waits for pages
to be come unlocked while releasing the mmap_sem.

The code also outlines what needs to be changed if we want to establish the
locking order as mmap_sem -> bo::reserve.

One slight issue that remains with this code is that the fault handler might
be prone to starvation if another thread countinously reserves the buffer.
IMO that usage pattern is highly unlikely.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>

Also check the busy placements before deciding to move a buffer object.
Failing to do this may result in a completely unneccessary move within a
single memory type.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NJakob Bornecrantz <jakob@vmware.com>
Cc: stable@vger.kernel.org

NO_EVICT bos that are not idle when all references are dropped are put on
the delayed destroy list. However, since they are not on LRU lists, they
are not available to shrinkers at that point, and buffers on the delayed
destroy list are not checked very often for idle.

So when these buffers are put on the delayed destroy list, clear the
NO_EVICT flag and put them on the right LRU list. This way they are
immediately available for eviction or shrinkers and will not cause false
OOMS.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NJakob Bornecrantz <jakob@vmware.com>

This helper is used only once and just wraps a call to
drm_vma_offset_add(). Remove this unneeded indirection to safe 10 lines of
code.
Signed-off-by: NDavid Herrmann <dh.herrmann@gmail.com>
Reviewed-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@gmail.com>

Instead of unmapping the nodes in TTM and GEM users manually, we provide
a generic wrapper which does the correct thing for all vma-nodes.

v2: remove bdev->dev_mapping test in ttm_bo_unmap_virtual_unlocked() as
ttm_mem_io_free_vm() does nothing in that case (io_reserved_vm is 0).
v4: Fix docbook comments
v5: use drm_vma_node_size()

Cc: Dave Airlie <airlied@redhat.com>
Cc: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Cc: Thomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDavid Herrmann <dh.herrmann@gmail.com>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NDave Airlie <airlied@gmail.com>