v2: fix even more
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NMonk.Liu <monk.liu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This marks the struct amdgpu_sched_ops const and
adjusts amd_sched_init to take a const pointer
for the ops param. The ops member of
struct amd_gpu_scheduler is also changed to const.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NNils Wallménius <nils.wallmenius@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

this is to fix fatal page fault error that occured if:
job is signaled/released after its timeout work is already
put to the global queue (in this case the cancel_delayed_work
will return false), which will lead to NX-protection error
page fault during job_timeout_func.
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Add two callbacks to scheduler to maintain jobs, and invoked for
job timeout calculations. Now TDR measures time gap from
job is processed by hw.

v2:
fix typo
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

original time out detect routine is incorrect, cuz it measures
the gap from job scheduled, but we should only measure the
gap from processed by hw.
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

the mirror_list will be used for later time out detect
feature.  This is needed to properly detect a GPU
timeout with the scheduler.
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

for those jobs submitted through scheduler, do not
free it immediately after scheduled, instead free it
in global workqueue by its sched fence signaling
callback function.

v2:
call uf's bo_undef after job_run()
call job's sync free after job_run()
no static inline __amdgpu_job_free() anymore, just use
kfree(job) to replace it.
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Consolidate job initialization in one place rather than
duplicating it in multiple places.
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

since the dependency job is also scheduled by the same
scheduler with the job depended on it, no need to
call wake up scheduler when the dep is scheduled.
Signed-off-by: NMonk Liu <Monk.Liu@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

umd somtimes will create a context for every ring,
that means some entities wouldn't be used at all.
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

As soon as we leave the spinlock after the job has been added to the job
queue, we can no longer rely on the job's data to be available.

I have seen a null-pointer dereference due to sched == NULL in
amd_sched_wakeup via amd_sched_entity_push_job and
amd_sched_ib_submit_kernel_helper. Since the latter initializes
sched_job->sched with the address of the ring scheduler, which is
guaranteed to be non-NULL, this race appears to be a likely culprit.
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Bugzilla: https://bugs.freedesktop.org/attachment.cgi?bugid=93079Reviewed-by: NChristian König <christian.koenig@amd.com>

Allows us to set priorities in the scheduler.
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>

As soon as we leave the spinlock after the job has been added to the job
queue, we can no longer rely on the job's data to be available.

I have seen a null-pointer dereference due to sched == NULL in
amd_sched_wakeup via amd_sched_entity_push_job and
amd_sched_ib_submit_kernel_helper. Since the latter initializes
sched_job->sched with the address of the ring scheduler, which is
guaranteed to be non-NULL, this race appears to be a likely culprit.
Signed-off-by: NNicolai Hähnle <nicolai.haehnle@amd.com>
Bugzilla: https://bugs.freedesktop.org/attachment.cgi?bugid=93079Reviewed-by: NChristian König <christian.koenig@amd.com>

This way the driver isn't limited in the dependency handling callback.

v2: remove extra check in amd_sched_entity_pop_job()
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>

We only need to wait for jobs to be scheduled when
the dependency is from the same scheduler.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>

Before this patch the scheduler fence was created when we push the job
into the queue, so we could only get the fence after pushing it.

The mutex now was necessary to prevent the thread pushing the jobs to
the hardware from running faster than the thread pushing the jobs into
the queue.

Otherwise the thread pushing jobs into the queue would have accessed
possible freed up memory when it tries to get a reference to the fence.

So what you get in the end is thread A:
mutex_lock(&job->lock);
...
Kick of thread B.
...
mutex_unlock(&job->lock);

And thread B:
mutex_lock(&job->lock);
....
mutex_unlock(&job->lock);
kfree(job);

I'm actually not sure if I'm still up to date on this, but this usage
pattern used to be not allowed with mutexes. See here as well
https://lwn.net/Articles/575460/.

v2: remove unrelated changes, fix missing owner
v3: rebased, add more commit message
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

The code was correct, but getting two references when the ownership
is linearly moved on is a bit awkward and just overhead.

Signed: Christian König <christian.koenig@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

OGL needs these tracepoints to investigate performance issue.

Change-Id: I5e58187d061253f7d665dfce8e4e163ba91d3e2b
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>

Change-Id: I45bb8ff10ef05dc3b15e31a77fbcf31117705f11
Signed-off-by: NChunming Zhou <David1.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>

cancel_delayed_work_sync is forbidden in interrupt context.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>

In two places amdgpu tries to tear down something it hasn't
initalised when failing. This is what happens when you
enable experimental support on topaz which then fails in
ring init.

This patch allows it to fail cleanly.

agd: Split out from from the original patch since the
scheduler is a driver independent.
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

We are going to submit them before the job anyway.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>

The first lockup fence will lock the fence list of scheduler.
Then cancel the delayed workqueues for all clean pending fences
without waiting the workqueues to finish.

Change-Id: I9bec826de1aa49d587b0662f3fb4a95333979429
Signed-off-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>

Change-Id: I67e987db0efdca28faa80b332b75571192130d33
Signed-off-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NDavid Zhou <david1.zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>

Embed the scheduler into the ring structure instead of allocating it.
Use the ring name directly instead of the id.

v2: rebased, whitespace cleanup
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: Chunming Zhou<david1.zhou@amd.com>

Just to be consistent with the other members.

v2: rename the ring member as well.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com> (v1)
Reviewed-by: Chunming Zhou<david1.zhou@amd.com>

Reorder the fields and properly return the kfifo_alloc error code.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: Chunming Zhou<david1.zhou@amd.com>

Use consistent naming across functions.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NDavid Zhou <david1.zhou@amd.com>
Signed-off-by: NJunwei Zhang <Jerry.Zhang@amd.com>

Just free the resources immediately after submitting the job.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>

And call the processed callback directly after submitting the job.

v2: split adding error handling into separate patch.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>

Otherwise the resource blocked by it will never be reclaimed.

v2: add DRM_ERROR.
v3: fix typo in commit message
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: Chunming Zhou<david1.zhou@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>

track sched job status like the length of job queue and hw job queue.

v2: fix build after rebase
Signed-off-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>

This should never happen so warn when the count does
not equal the expected size.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

with interruptible, the push_job maybe return -ERESTARTSYS,
then result in push_job error.

E.g. bug trace:
[  181.618860] *****amdgpu_copy_buffer:fence->seq:0x0000000048d8758b, contxt:1207959552, ref:683967304, r:-512
[  181.618929] BUG: unable to handle kernel paging request at ffffffff811aa266
[  181.625887] IP: [<ffffffff81548ffc>] reservation_object_add_excl_fence+0x3c/0x120
...
[  181.859767]  [<ffffffff811aa266>] ? unmap_mapping_range+0x66/0x110
[  181.865928]  [<ffffffffc0608ac1>] ttm_bo_move_accel_cleanup+0x41/0x3c0 [ttm]
[  181.872971]  [<ffffffffc062d382>] amdgpu_move_blit.isra.18+0x122/0x150 [amdgpu]
[  181.880254]  [<ffffffff811aa266>] ? unmap_mapping_range+0x66/0x110
[  181.886420]  [<ffffffffc062d709>] amdgpu_bo_move+0xa9/0x200 [amdgpu]
[  181.892753]  [<ffffffffc0606e8d>] ttm_bo_handle_move_mem+0x26d/0x5c0 [ttm]
Signed-off-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>

This way the scheduler doesn't wait in it's work thread any more.

v2: fix race conditions
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>

v2: fix another race condition
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>

Freeing up a queue after signalling it isn't race free.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>

Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>

Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>

Removing the entity from scheduling can deadlock the whole system.
Wait forever till the remaining IBs are scheduled.

v2: fix comment as well
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Reviewed-by: NJammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com> (v1)