Setting the no_gpu_wait flag means that the allocate BO must be available
immediately and we can't wait for any GPU operation to finish.

v2: squash in mem leak fix, rebase
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Be more consistent with the naming of the other DMA-buf objects.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/323401/

All users moved to ttm_buffer_object->base.resv
Signed-off-by: NGerd Hoffmann <kraxel@redhat.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190805140119.7337-18-kraxel@redhat.com

Signed-off-by: NGerd Hoffmann <kraxel@redhat.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190805140119.7337-11-kraxel@redhat.com

Initialize both ttm_buffer_object->resv and ttm_buffer_object->base.resv
pointers.  This allows to move users from the former to the latter.  When
all users are moved we can drop ttm_buffer_object->resv.
Signed-off-by: NGerd Hoffmann <kraxel@redhat.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190805140119.7337-10-kraxel@redhat.com

Drop vma_node from ttm_buffer_object, use the gem struct
(base.vma_node) instead.
Signed-off-by: NGerd Hoffmann <kraxel@redhat.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190805140119.7337-9-kraxel@redhat.com

Drop ttm_resv from ttm_buffer_object, use the gem reservation object
(base._resv) instead.
Signed-off-by: NGerd Hoffmann <kraxel@redhat.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190805140119.7337-8-kraxel@redhat.com

Complete the abstraction of the ww_mutex inside the reservation object.

This allows us to add more handling and debugging to the reservation
object in the future.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/320761/

This notifies the driver that a BO is about to be released.

Releasing a BO also invokes the move_notify callback from
ttm_bo_cleanup_memtype_use, but that happens too late for anything
that would add fences to the BO and require a delayed delete.
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This reverts commit 031e610a, reversing
changes made to 52d2d44e.

The mm changes in there we premature and not fully ack or reviewed by core mm folks,
I dropped the ball by merging them via this tree, so lets take em all back out.
Signed-off-by: NDave Airlie <airlied@redhat.com>

Returning -EAGAIN prevents ttm_bo_mem_space from trying alternate
placements and can lead to live-locks in amdgpu_cs, retrying
indefinitely and never succeeding.

Fixes: d367bd2a ("drm/ttm: fix busy memory to fail other user v10")
Signed-off-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add a pointer to the struct vm_operations_struct in the bo_device, and
assign that pointer to the default value currently used.

The driver can then optionally modify that pointer and the new value
can be used for each new vma created.

Cc: "Christian König" <christian.koenig@amd.com>
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>

BOs on the LRU might be blocked during command submission
and cause OOM situations.

Avoid this by blocking for the first busy BO not locked by
the same ticket as the BO we are searching space for.

v10: completely start over with the patch since we didn't
     handled a whole bunch of corner cases.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Tested-by: NPierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Move BOs which are currently in a lower domain to the new
LRU before allocating backing space while validating.

This makes sure that we always have enough entries on the
LRU to allow for other processes to wait for an operation
to complete.

v2: generalize the test
v3: fix rebase error
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Tested-by: NPierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We tried this once before, but that turned out to be more
complicated than thought. With all the right prerequisites
it looks like we can do this now.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NChunming Zhou <david1.zhou@amd.com>
Tested-by: NPierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

If drivers don't prefer a system memory placement
they should not but it into the placement list first.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NChunming Zhou <david1.zhou@amd.com>
Tested-by: NPierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

When a signal arrives we should return immediately for
handling it and not try other placements first.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NChunming Zhou <david1.zhou@amd.com>
Tested-by: NPierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We are already doing this for DMA-buf imports and also for
amdgpu VM BOs for quite a while now.

If this doesn't run into any problems we are probably going
to stop removing BOs from the LRU altogether.

v2: drop BUG_ON from ttm_bo_add_to_lru
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Tested-by: NPierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

When a driver unloads without unloading TTM we don't correctly
clear the global structures leading to errors on re-init.

Next step should probably be to remove the global structures and
kobjs all together, but this is tricky since we need to maintain
backward compatibility.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NKarol Herbst <kherbst@redhat.com>
Tested-by: NKarol Herbst <kherbst@redhat.com>
CC: stable@vger.kernel.org # 5.0.x
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

the ttm_bo_add_move_fence takes a reference to the struct dma_fence, but
failed to release it on the error path, leading to a memory leak.
add dma_fence_put before return when error occur.
Signed-off-by: NLin Yi <teroincn@163.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

GEM defines DRM_FILE_PAGE_OFFSET_{START,SIZE} constants for the
mmap-able range of addresses. TTM can use them as well.
Signed-off-by: NThomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Acked-by: NHans de Goede <hdegoede@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The parameter file_page_offset is a constant shared by all drivers. Just
replace it with the constant itself.
Signed-off-by: NThomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Acked-by: NHans de Goede <hdegoede@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Both functions are obsolete and all calls have been replaced by
ttm_bo_get and ttm_bo_put.
Signed-off-by: NThomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

allow driver do somethings when lru changed.
v2:
address Michel's comments.
Signed-off-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

No need for pr_err here, the pr_err message in ttm_bo_evict is enough
to draw attention to something not going as planned.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

ttm_bo_glob and ttm_mem_glob are defined as structure instance, while
not allocated by kzalloc, so kfree should not be invoked to release
them anymore. Otherwise, it will cause the following kernel BUG when
unloading amdgpu module

[   48.419294] kernel BUG at /build/linux-5s7Xkn/linux-4.15.0/mm/slub.c:3894!
[   48.419352] invalid opcode: 0000 [#1] SMP PTI
[   48.419387] Modules linked in: amdgpu(OE-) amdchash(OE) amdttm(OE) amd_sched(OE) amdkcl(OE) amd_iommu_v2 drm_kms_helper drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi pcbc snd_seq snd_seq_device snd_timer aesni_intel snd soundcore joydev aes_x86_64 crypto_simd glue_helper cryptd input_leds mac_hid serio_raw binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 8139too psmouse i2c_piix4 8139cp mii floppy pata_acpi
[   48.419782] CPU: 1 PID: 1281 Comm: modprobe Tainted: G           OE    4.15.0-20-generic #21-Ubuntu
[   48.419838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[   48.419901] RIP: 0010:kfree+0x137/0x180
[   48.419934] RSP: 0018:ffffb02101273bf8 EFLAGS: 00010246
[   48.419974] RAX: ffffeee1418ad7e0 RBX: ffffffffc075f100 RCX: ffff8fed7fca7ed0
[   48.420025] RDX: 0000000000000000 RSI: 000000000003440e RDI: 0000000022400000
[   48.420073] RBP: ffffb02101273c10 R08: 0000000000000010 R09: ffff8fed7ffd3680
[   48.420121] R10: ffffeee1418ad7c0 R11: ffff8fed7ffd3000 R12: ffffffffc075e2c0
[   48.420169] R13: ffffffffc074ec10 R14: ffff8fed73063900 R15: ffff8fed737428e8
[   48.420216] FS:  00007fdc912ec540(0000) GS:ffff8fed7fc80000(0000) knlGS:0000000000000000
[   48.420267] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   48.420308] CR2: 000055fa40c30060 CR3: 000000023470a006 CR4: 00000000003606e0
[   48.420358] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   48.420405] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   48.420452] Call Trace:
[   48.420485]  ttm_bo_global_kobj_release+0x20/0x30 [amdttm]
[   48.420528]  kobject_release+0x6a/0x180
[   48.420562]  kobject_put+0x28/0x50
[   48.420595]  ttm_bo_global_release+0x36/0x50 [amdttm]
[   48.420636]  amdttm_bo_device_release+0x119/0x180 [amdttm]
[   48.420678]  ? amdttm_bo_clean_mm+0xa6/0xf0 [amdttm]
[   48.420760]  amdgpu_ttm_fini+0xc9/0x180 [amdgpu]
[   48.420821]  amdgpu_bo_fini+0x12/0x40 [amdgpu]
[   48.420889]  gmc_v9_0_sw_fini+0x40/0x50 [amdgpu]
[   48.420947]  amdgpu_device_fini+0x36f/0x4c0 [amdgpu]
[   48.421007]  amdgpu_driver_unload_kms+0xb4/0x150 [amdgpu]
[   48.421058]  drm_dev_unregister+0x46/0xf0 [drm]
[   48.421102]  drm_dev_unplug+0x12/0x70 [drm]
Signed-off-by: NTrigger Huang <Trigger.Huang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Make sure that the global BO state is always correctly initialized.

This allows removing all the device code to initialize it.

v2: fix up vbox (Alex)
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

As the name says we only need one global instance of ttm_bo_global.

Just use a single exported instance which is save to initialize multiple times.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This way it can protect the whole BO global state.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

As the name says we only need one global instance of ttm_mem_global.

Drop all the driver initialization and just use a single exported
instance which is initialized during BO global initialization.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

So far, struct ttm_bo_global_ref was the only way of initializing a struct
ttm_bo_global. Providing separate initializer and release functions for
struct ttm_bo_global gives drivers the option of implementing their own
init and release callbacks for drm_global_references of type
DRM_GLOBAL_TTM_BO.

The original functions for initializing and releasing via struct
ttm_bo_global_ref are wrappers around the new interfaces.
Signed-off-by: NThomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The functions ttm_bo_global_init() and ttm_bo_global_release() do not
receive an argument of type struct ttm_bo_global. Both take a struct
drm_global_reference that contains points to a struct ttm_bo_global_ref.
Renaming them reflects this.
Signed-off-by: NThomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Let's support simultaneous submissions to multiple engines.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NMichel Dänzer <michel.daenzer@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Link: https://patchwork.kernel.org/patch/10626149/

Move all entries between @first and including @last before @head.

This is useful for LRU lists where a whole block of entries should be
moved to the end of the list.

Used as a band aid in TTM, but better placed in the common list headers.
Acked-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

While cutting the lists we sometimes accidentally added a list_head from
the stack to the LRUs, effectively corrupting the list.

Remove the list cutting and use explicit list manipulation instead.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-and-Tested: Huang Rui <ray.huang@amd.com>
Tested-by: NMike Lothian <mike@fireburn.co.uk>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Staring at the function for six hours, just to essentially move one line
of code. The problem was that the first list_cut_position call could result
in list2 pointing to la-la-land.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Tested-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The first parameter of list_cut_position() must point to an initialized
list.

Noticed thanks to KASAN pointing out something's fishy here.

Fixes: "drm/ttm: add bulk move function on LRU"
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NMichel Dänzer <michel.daenzer@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This function allow us to bulk move a group of BOs to the tail of their LRU.
The positions of group of BOs are stored on the (first, last) bulk_move_pos
structure.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NHuang Rui <ray.huang@amd.com>
Tested-by: NMike Lothian <mike@fireburn.co.uk>
Tested-by: NDieter Nützel <Dieter@nuetzel-hh.de>
Acked-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

When move a BO to the end of LRU, it need remember the BO positions.
Make sure all moved bo in between "first" and "last". And they will be bulk
moving together.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NHuang Rui <ray.huang@amd.com>
Tested-by: NMike Lothian <mike@fireburn.co.uk>
Tested-by: NDieter Nützel <Dieter@nuetzel-hh.de>
Acked-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

A call to ttm_bo_unref() clears the supplied pointer to NULL, while
ttm_bo_put() does not. None of the converted call sites requires the
pointer to become NULL, so the respective assign operations has been
left out from the patch.
Signed-off-by: NThomas Zimmermann <contact@tzimmermann.org>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>