It's used in ci_dpm.c so move it there and make it static.
Reviewed-by: NRodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Highly concurrent Piglit runs can trigger a race condition where a pending
SDMA job on a buffer object is never executed because the corresponding
process is killed (perhaps due to a crash). Since the job's fences were
never signaled, the buffer object was effectively leaked. Worse, the
buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with
kernel stacks like:

    [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
    [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
    [<ffffffffbc5e82d2>] reservation_object_wait_timeout_rcu+0x1c2/0x300
    [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
    [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
    [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
    [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
    [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
    [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
    [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
    [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
    [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
    [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
    [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
    [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
    [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
    [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
    [<ffffffffffffffff>] 0xffffffffffffffff

Note: The correctness of this change depends on the earlier commit
"drm/amd/sched: move adding finish callback to amd_sched_job_begin"

v2: set an error on the finished fence
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

amd_sched_process_job drops the fence reference, so NULL out the s_fence
field before adding it as a callback to guard against accidentally using
s_fence after it may have be freed.

v2: add a clarifying comment
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The finish callback is responsible for removing the job from the ring
mirror list, among other things. It makes sense to add it as callback
in the place where the job is added to the ring mirror list.
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The function does not actually remove the job from the FIFO, so "peek"
describes it better.
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Fix two minor 80 char issues.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Try to allocate huge pages when it makes sense.

v2: fix comment and use ifdef
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Correctly handle different page sizes in the memory accounting.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Nobody is actually using that, remove it.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add UVD encode IRQ handle and enable the UVD encode trap
Signed-off-by: NJames Zhu <James.Zhu@amd.com>
Reviewed-and-Tested-by: NLeo Liu <leo.liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Generate create/destroy messages to test UVD encode indirect buffer function.
And enable UVD encode IB test during device initialization.
Signed-off-by: NJames Zhu <James.Zhu@amd.com>
Reviewed-and-Tested-by: NLeo Liu <leo.liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add UVD encode ring test functions. And enable UVD encode ring test
during UVD encode hardware initialization.
Signed-off-by: NJames Zhu <James.Zhu@amd.com>
Reviewed-and-Tested-by: NLeo Liu <leo.liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add UVD encode ring vm functions to handle frame ecoding.

v2: squash in warning fix (James)
Signed-off-by: NJames Zhu <James.Zhu@amd.com>
Reviewed-and-Tested-by: NLeo Liu <leo.liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NJames Zhu <James.Zhu@amd.com>
Reviewed-and-Tested-by: NLeo Liu <leo.liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

UVD 6.3 has two UVD encode rings.  Add the ring structures and initialize the hw ring buffers.
Currently only ASIC Polaris10/11/12 uses UVD6.3 encode engine on HEVC encoding.
Signed-off-by: NJames Zhu <James.Zhu@amd.com>
Reviewed-and-Tested-by: NLeo Liu <leo.liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add new UVD encode ring methods get/set/emit/flush/sync to support uvd6.3 HEVC encoding
Signed-off-by: NJames Zhu <James.Zhu@amd.com>
Reviewed-and-Tested-by: NLeo Liu <leo.liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add UVD encode command interface definition for uvd6.3 HEVC encoding
Signed-off-by: NJames Zhu <James.Zhu@amd.com>
Reviewed-and-Tested-by: NLeo Liu <leo.liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add UVD encode write/read/size/base registers definition for uvd6.3 HEVC ecoding
Signed-off-by: NJames Zhu <James.Zhu@amd.com>
Reviewed-and-Tested-by: NLeo Liu <leo.liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

regression issue caused by
commit 47047263
("drm/amd/powerplay: delete eventmgr related files.")
Reviewed-by: NEvan Quan <evan.quan@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This partially reverts 0b6b4cbf77c995a34a4ec3d705a636434dadc51a and fixes
the noise issues on Tonga.
Signed-off-by: NTom St Denis <tom.stdenis@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Acked-by: NHarry Wentland <harry.wentland@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NEvan Quan <evan.quan@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

v2: squash in rebase fix (Tom)
Signed-off-by: NEvan Quan <evan.quan@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

v2: squash in rebase fix (Tom)
Signed-off-by: NEvan Quan <evan.quan@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

v2: squash in rebase fix (Tom)
Signed-off-by: NEvan Quan <evan.quan@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NEvan Quan <evan.quan@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

v2: squash in typo fix (Tom)
Signed-off-by: NEvan Quan <evan.quan@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The dead circular lock senario captured is as followed.
The idea of the fix is moving read_user_wptr outside of
acquire_queue...release_queue critical section

[   63.477482] WARNING: possible circular locking dependency detected
[   63.484091] 4.12.0-kfd-ozeng #3 Not tainted
[   63.488531] ------------------------------------------------------
[   63.495146] HelloWorldLoop/2526 is trying to acquire lock:
[   63.501011]  (&mm->mmap_sem){++++++}, at: [<ffffffff911898ce>] __might_fault+0x3e/0x90
[   63.509472]
               but task is already holding lock:
[   63.515716]  (&adev->srbm_mutex){+.+...}, at: [<ffffffffc0484feb>] lock_srbm+0x2b/0x50 [amdgpu]
[   63.525099]
               which lock already depends on the new lock.

[   63.533841]
               the existing dependency chain (in reverse order) is:
[   63.541839]
               -> #2 (&adev->srbm_mutex){+.+...}:
[   63.548178]        lock_acquire+0x6d/0x90
[   63.552461]        __mutex_lock+0x70/0x8c0
[   63.556826]        mutex_lock_nested+0x16/0x20
[   63.561603]        gfx_v8_0_kiq_resume+0x1039/0x14a0 [amdgpu]
[   63.567817]        gfx_v8_0_hw_init+0x204d/0x2210 [amdgpu]
[   63.573675]        amdgpu_device_init+0xdea/0x1790 [amdgpu]
[   63.579640]        amdgpu_driver_load_kms+0x63/0x220 [amdgpu]
[   63.585743]        drm_dev_register+0x145/0x1e0
[   63.590605]        amdgpu_pci_probe+0x11e/0x160 [amdgpu]
[   63.596266]        local_pci_probe+0x40/0xa0
[   63.600803]        pci_device_probe+0x134/0x150
[   63.605650]        driver_probe_device+0x2a1/0x460
[   63.610785]        __driver_attach+0xdc/0xe0
[   63.615321]        bus_for_each_dev+0x5f/0x90
[   63.619984]        driver_attach+0x19/0x20
[   63.624337]        bus_add_driver+0x40/0x270
[   63.628908]        driver_register+0x5b/0xe0
[   63.633446]        __pci_register_driver+0x5b/0x60
[   63.638586]        rtsx_pci_switch_output_voltage+0x1d/0x20 [rtsx_pci]
[   63.645564]        do_one_initcall+0x4c/0x1b0
[   63.650205]        do_init_module+0x56/0x1ea
[   63.654767]        load_module+0x208c/0x27d0
[   63.659335]        SYSC_finit_module+0x96/0xd0
[   63.664058]        SyS_finit_module+0x9/0x10
[   63.668629]        entry_SYSCALL_64_fastpath+0x1f/0xbe
[   63.674088]
               -> #1 (reservation_ww_class_mutex){+.+.+.}:
[   63.681257]        lock_acquire+0x6d/0x90
[   63.685551]        __ww_mutex_lock.constprop.11+0x8c/0xed0
[   63.691426]        ww_mutex_lock+0x67/0x70
[   63.695802]        amdgpu_verify_access+0x6d/0x100 [amdgpu]
[   63.701743]        ttm_bo_mmap+0x8e/0x100 [ttm]
[   63.706615]        amdgpu_bo_mmap+0xd/0x60 [amdgpu]
[   63.711814]        amdgpu_mmap+0x35/0x40 [amdgpu]
[   63.716904]        mmap_region+0x3b5/0x5a0
[   63.721255]        do_mmap+0x400/0x4d0
[   63.725260]        vm_mmap_pgoff+0xb0/0xf0
[   63.729625]        SyS_mmap_pgoff+0x19e/0x260
[   63.734292]        SyS_mmap+0x1d/0x20
[   63.738199]        entry_SYSCALL_64_fastpath+0x1f/0xbe
[   63.743681]
               -> #0 (&mm->mmap_sem){++++++}:
[   63.749641]        __lock_acquire+0x1401/0x1420
[   63.754491]        lock_acquire+0x6d/0x90
[   63.758750]        __might_fault+0x6b/0x90
[   63.763176]        kgd_hqd_load+0x24f/0x270 [amdgpu]
[   63.768432]        load_mqd+0x4b/0x50 [amdkfd]
[   63.773192]        create_queue_nocpsch+0x535/0x620 [amdkfd]
[   63.779237]        pqm_create_queue+0x34d/0x4f0 [amdkfd]
[   63.784835]        kfd_ioctl_create_queue+0x282/0x670 [amdkfd]
[   63.790973]        kfd_ioctl+0x310/0x4d0 [amdkfd]
[   63.795944]        do_vfs_ioctl+0x90/0x6e0
[   63.800268]        SyS_ioctl+0x74/0x80
[   63.804207]        entry_SYSCALL_64_fastpath+0x1f/0xbe
[   63.809607]
               other info that might help us debug this:

[   63.818026] Chain exists of:
                 &mm->mmap_sem --> reservation_ww_class_mutex --> &adev->srbm_mutex

[   63.830382]  Possible unsafe locking scenario:

[   63.836605]        CPU0                    CPU1
[   63.841364]        ----                    ----
[   63.846123]   lock(&adev->srbm_mutex);
[   63.850061]                                lock(reservation_ww_class_mutex);
[   63.857475]                                lock(&adev->srbm_mutex);
[   63.864084]   lock(&mm->mmap_sem);
[   63.867657]
                *** DEADLOCK ***

[   63.873884] 3 locks held by HelloWorldLoop/2526:
[   63.878739]  #0:  (&process->mutex){+.+.+.}, at: [<ffffffffc06e1a9a>] kfd_ioctl_create_queue+0x24a/0x670 [amdkfd]
[   63.889543]  #1:  (&dqm->lock){+.+...}, at: [<ffffffffc06eedeb>] create_queue_nocpsch+0x3b/0x620 [amdkfd]
[   63.899684]  #2:  (&adev->srbm_mutex){+.+...}, at: [<ffffffffc0484feb>] lock_srbm+0x2b/0x50 [amdgpu]
[   63.909500]
               stack backtrace:
[   63.914187] CPU: 3 PID: 2526 Comm: HelloWorldLoop Not tainted 4.12.0-kfd-ozeng #3
[   63.922184] Hardware name: AMD Carrizo/Gardenia, BIOS WGA5819N_Weekly_15_08_1 08/19/2015
[   63.930865] Call Trace:
[   63.933464]  dump_stack+0x85/0xc9
[   63.936999]  print_circular_bug+0x1f9/0x207
[   63.941442]  __lock_acquire+0x1401/0x1420
[   63.945745]  ? lock_srbm+0x2b/0x50 [amdgpu]
[   63.950185]  lock_acquire+0x6d/0x90
[   63.953885]  ? __might_fault+0x3e/0x90
[   63.957899]  __might_fault+0x6b/0x90
[   63.961699]  ? __might_fault+0x3e/0x90
[   63.965755]  kgd_hqd_load+0x24f/0x270 [amdgpu]
[   63.970577]  load_mqd+0x4b/0x50 [amdkfd]
[   63.974745]  create_queue_nocpsch+0x535/0x620 [amdkfd]
[   63.980242]  pqm_create_queue+0x34d/0x4f0 [amdkfd]
[   63.985320]  kfd_ioctl_create_queue+0x282/0x670 [amdkfd]
[   63.991021]  kfd_ioctl+0x310/0x4d0 [amdkfd]
[   63.995499]  ? kfd_ioctl_destroy_queue+0x70/0x70 [amdkfd]
[   64.001234]  do_vfs_ioctl+0x90/0x6e0
[   64.005065]  ? up_read+0x1a/0x40
[   64.008496]  SyS_ioctl+0x74/0x80
[   64.011955]  entry_SYSCALL_64_fastpath+0x1f/0xbe
[   64.016863] RIP: 0033:0x7f4b3bd35f07
[   64.020696] RSP: 002b:00007ffe7689ec38 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[   64.028786] RAX: ffffffffffffffda RBX: 00000000002a2000 RCX: 00007f4b3bd35f07
[   64.036414] RDX: 00007ffe7689ecb0 RSI: 00000000c0584b02 RDI: 0000000000000005
[   64.044045] RBP: 00007f4a3212d000 R08: 00007f4b3c919000 R09: 0000000000080000
[   64.051674] R10: 00007f4b376b64b8 R11: 0000000000000246 R12: 00007f4a3212d000
[   64.059324] R13: 0000000000000015 R14: 0000000000000064 R15: 00007ffe7689ef50
Signed-off-by: NOak Zeng <Oak.Zeng@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The functions alloc_pasid  and free_pasid are local to the
source and do not need to be in global scope, so make them static.

Cleans up sparse warnings:
warning: symbol 'alloc_pasid' was not declared. Should it be static?
warning: symbol 'free_pasid' was not declared. Should it be static?
Signed-off-by: NColin Ian King <colin.king@canonical.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

for being able to convert an amdgpu fence into one of the handles.
Mesa will use this.
Reviewed-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NMarek Olšák <marek.olsak@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Reviewed-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NMarek Olšák <marek.olsak@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

For amdgpu.

drm_syncobj_create is renamed to drm_syncobj_create_as_handle, and new
helpers drm_syncobj_create and drm_syncobj_get_handle are added.
Reviewed-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NMarek Olšák <marek.olsak@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

don't need to check pp_valid, all pp
export functions are moved to ip_funcs
and pp_funcs. so just need to check the
function point.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

v2: squash in regression fix (Rex)
v3: Squash in regression fix (Rex)
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

1. use flag PP_DPM_DISABLED within powerplay
   notify amdgpu dpm state by cgs interface.
2. delete redundant virtualization check in
   powerplay

v2: squash in fix for hwmgr_init (Rex)
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Along the same lines as rewriting the polaris code, this rewrites
the fiji code, and reduces the driver size by ~40k.
Signed-off-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This reduces the pwrvirus table size by 30k, by moving the
sequences of writes to the data register into blocks.
Signed-off-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This removes the init path as well, since the init path
just did some constant init of some structs.
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

These don't seem to change at runtime, and the initialisers
are constant data. This could be improved by not selecting
the apu/non-apu path on each pcie read/write access.
Signed-off-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>