Check if 'aspace' is set before using it as it will stay null without
IOMMU, such as on msm8974.

Fixes: bc211258 ("drm/msm/gpu: Track global faults per address-space")
Signed-off-by: NLuca Weiss <luca@z3ntu.xyz>
Link: https://lore.kernel.org/r/20220421203455.313523-1-luca@z3ntu.xyzSigned-off-by: NRob Clark <robdclark@chromium.org>

The ring seqno counter duplicates the fence-context last_fence counter.
They end up getting incremented in lock-step, on the same scheduler
thread, but the split just makes things less obvious.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Link: https://lore.kernel.org/r/20220411215849.297838-3-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

In the cause of using the GPU via virtgpu, the host side process is
really a sort of proxy, and not terribly interesting from the PoV of
crash/fault logging.  Add a way to override these per process so that
we can see the guest process's name.

v2: Handle kmalloc failure, add comment to explain kstrdup returns
    NULL if passed NULL [Dan Carpenter]
Signed-off-by: NRob Clark <robdclark@chromium.org>
Link: https://lore.kernel.org/r/20220317165144.222101-4-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

Deduplicate this from fault_worker and recover_worker.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Link: https://lore.kernel.org/r/20220317165144.222101-3-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

Add a SYSPROF param for system profiling tools like Mesa's pps-producer
(perfetto) to control behavior related to system-wide performance
counter collection.  In particular, for profiling, one wants to ensure
that GPU context switches do not effect perfcounter state, and might
want to suppress suspend (which would cause counters to lose state).

v2: Swap the order in msm_file_private_set_sysprof() [sboyd] and
    initialize the sysprof_active refcount to one (because the under/
    overflow checking in refcount_t doesn't expect a 0->1 transition)
    meaning that values greater than 1 means sysprof is active.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Link: https://lore.kernel.org/r/20220304005317.776110-4-robdclark@gmail.com

Use generic name for resources like irq and kthread instead of hardware
specific name to make it easier to grep.
Signed-off-by: NAkhil P Oommen <quic_akhilpo@quicinc.com>
Link: https://lore.kernel.org/r/20220226005021.v2.1.Id3d2e7391192c86d0783aeb307d3f9fb61f9efee@changeidSigned-off-by: NRob Clark <robdclark@chromium.org>

Other processes don't need to know about faults that they are isolated
from by virtue of address space isolation.  They are only interested in
whether some of their state might have been corrupted.

But to be safe, also track unattributed faults.  This case should really
never happen unless there is a kernel bug (and that would never happen,
right?)

v2: Instead of adding a new param, just change the behavior of the
    existing param to match what userspace actually wants [anholt]

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/5934Signed-off-by: NRob Clark <robdclark@chromium.org>
Link: https://lore.kernel.org/r/20220201161618.778455-3-robdclark@gmail.comReviewed-by: NEmma Anholt <emma@anholt.net>
Signed-off-by: NRob Clark <robdclark@chromium.org>

msm_ioremap() functions take additional argument dbgname which is now
unused.
Signed-off-by: NDmitry Baryshkov <dmitry.baryshkov@linaro.org>
Reviewed-by: NStephen Boyd <swboyd@chromium.org>
Link: https://lore.kernel.org/r/20220105232700.444170-2-dmitry.baryshkov@linaro.orgSigned-off-by: NDmitry Baryshkov <dmitry.baryshkov@linaro.org>

System suspend uses pm_runtime_force_suspend(), which cheekily bypasses
the runpm reference counts.  This doesn't actually work so well when the
GPU is active.  So add a reasonable delay waiting for the GPU to become
idle.

Alternatively we could just return -EBUSY in this case, but that has the
disadvantage of causing system suspend to fail.

v2: s/ret/remaining [sboyd], and switch to using active_submits count
    to ensure we aren't racing with submit cleanup (and devfreq idle
    work getting scheduled, etc)
v3: fix inverted logic
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NBjorn Andersson <bjorn.andersson@linaro.org>
Reviewed-by: NAngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
Link: https://lore.kernel.org/r/20220108180913.814448-2-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

Add some helpers for fence comparision, which handle rollover properly,
and stop open coding fence seqno comparisions.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NAkhil P Oommen <akhilpo@codeaurora.org>
Link: https://lore.kernel.org/r/20211109181117.591148-5-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

The remaining struct_mutex usage is just to serialize various gpu
related things (submit/retire/recover/fault/etc), so replace
struct_mutex with gpu->lock.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Link: https://lore.kernel.org/r/20211109181117.591148-4-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

cur_ctx_seqno already does the same thing, but handles the edge cases
where a refcnt'd context can live after lastclose.  So let's not have
two ways to do the same thing.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NAkhil P Oommen <akhilpo@codeaurora.org>
Link: https://lore.kernel.org/r/20211109181117.591148-3-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

Coverity complains of a possible NULL dereference:

CID 120718 (#1 of 1): Dereference null return value (NULL_RETURNS)
23. dereference: Dereferencing a pointer that might be NULL state->bos when
    calling msm_gpu_crashstate_get_bo. [show details]
301                        msm_gpu_crashstate_get_bo(state, submit->bos[i].obj,
302                                submit->bos[i].iova, submit->bos[i].flags);

Fix this by employing the same state->bos NULL check as is used in the next
for loop.

Cc: Rob Clark <robdclark@gmail.com>
Cc: Sean Paul <sean@poorly.run>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: linux-arm-msm@vger.kernel.org
Cc: dri-devel@lists.freedesktop.org
Cc: freedreno@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: NTim Gardner <tim.gardner@canonical.com>
Reviewed-by: NDmitry Baryshkov <dmitry.baryshkov@linaro.org>
Link: https://lore.kernel.org/r/20210929162554.14295-1-tim.gardner@canonical.comSigned-off-by: NDmitry Baryshkov <dmitry.baryshkov@linaro.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

For existing adrenos, there is one or more ringbuffer, depending on
whether preemption is supported.  When preemption is supported, each
ringbuffer has it's own priority.  A submitqueue (which maps to a
gl context or vk queue in userspace) is mapped to a specific ring-
buffer at creation time, based on the submitqueue's priority.

Each ringbuffer has it's own drm_gpu_scheduler.  Each submitqueue
maps to a drm_sched_entity.  And each submit maps to a drm_sched_job.

Closes: https://gitlab.freedesktop.org/drm/msm/-/issues/4Signed-off-by: NRob Clark <robdclark@chromium.org>
Acked-by: NChristian König <christian.koenig@amd.com>
Link: https://lore.kernel.org/r/20210728010632.2633470-10-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

Move all the locked/active/pinned state handling to msm_gem_submit.c.
In particular, for drm/scheduler, we'll need to do all this before
pushing the submit job to the scheduler.  But while we're at it we can
get rid of the dupicate pin and refcnt.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Acked-by: NChristian König <christian.koenig@amd.com>
Link: https://lore.kernel.org/r/20210728010632.2633470-7-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

No idea why we were still using this.  It certainly hasn't been needed
for some time.  So drop the pointless twin codepaths.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Acked-by: NChristian König <christian.koenig@amd.com>
Link: https://lore.kernel.org/r/20210728010632.2633470-4-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

This adds a few things to try and make frequency scaling better match
the workload:

1) Longer polling interval to avoid whip-lashing between too-high and
   too-low frequencies in certain workloads, like mobile games which
   throttle themselves to 30fps.

   Previously our polling interval was short enough to let things
   ramp down to minimum freq in the "off" frame, but long enough to
   not react quickly enough when rendering started on the next frame,
   leading to uneven frame times.  (Ie. rather than a consistent 33ms
   it would alternate between 16/33/48ms.)

2) Awareness of when the GPU is active vs idle.  Since we know when
   the GPU is active vs idle, we can clamp the frequency down to the
   minimum while it is idle.  (If it is idle for long enough, then
   the autosuspend delay will eventually kick in and power down the
   GPU.)

   Since devfreq has no knowledge of powered-but-idle, this takes a
   small bit of trickery to maintain a "fake" frequency while idle.
   This, combined with the longer polling period allows devfreq to
   arrive at a reasonable "active" frequency, while still clamping
   to minimum freq when idle to reduce power draw.

3) Boost.  Because simple_ondemand needs to see a certain threshold
   of busyness to ramp up, we could end up needing multiple polling
   cycles before it reacts appropriately on interactive workloads
   (ex. scrolling a web page after reading for some time), on top
   of the already lengthened polling interval, when we see a idle
   to active transition after a period of idle time we boost the
   frequency that we return to.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Link: https://lore.kernel.org/r/20210726144653.2180096-4-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

Before we start adding more cleverness, split it into it's own file.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NDmitry Baryshkov <dmitry.baryshkov@linaro.org>
Link: https://lore.kernel.org/r/20210726144653.2180096-2-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

Nothing we do to in update_fences() can't be done in an atomic context,
so move this into the GPU's irq context to reduce latency (and call
dma_fence_signal() so we aren't relying on dma_fence_is_signaled() which
would defeat the purpose).
Signed-off-by: NRob Clark <robdclark@chromium.org>
Link: https://lore.kernel.org/r/20210726144359.2179302-3-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

Wire up support to stall the SMMU on iova fault, and collect a devcore-
dump snapshot for easier debugging of faults.

Currently this is a6xx-only, but mostly only because so far it is the
only one using adreno-smmu-priv.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Acked-by: NJordan Crouse <jordan@cosmicpenguin.net>
Link: https://lore.kernel.org/r/20210610214431.539029-6-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

While keeping the previous default value for hangcheck period,
we allow now the possibility of configuring its value via
debugfs.
Signed-off-by: NSamuel Iglesias Gonsalvez <siglesias@igalia.com>
Link: https://lore.kernel.org/r/20210607104441.184700-1-siglesias@igalia.comSigned-off-by: NRob Clark <robdclark@chromium.org>

Performance counts, and ALWAYS_ON counters used for capturing GPU
timestamps, lose their state across suspend/resume cycles.  Userspace
tooling for performance monitoring needs to be aware of this.  For
example, after a suspend userspace needs to recalibrate it's offset
between CPU and GPU time.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Acked-by: NJordan Crouse <jordan@cosmicpenguin.net>
Link: https://lore.kernel.org/r/20210325012358.1759770-3-robdclark@gmail.comSigned-off-by: NRob Clark <robdclark@chromium.org>

Previously we only held obj lock in the _active_get() path, and relied
on atomic_dec_return() to not be racy in the _active_put() path where
obj lock was not held.

But this is a false sense of security.  Unlike obj lifetime refcnt,
where you do not expect to *increase* the refcnt after the last put
(which would mean that something has gone horribly wrong with the
object liveness reference counting), the active_count can increase
again from zero.  Racing _active_put()s and _active_get()s could leave
the obj on the wrong mm list.

But in the retire path, immediately after the _active_put(), the
_unpin_iova() would acquire obj lock.  So just move the locking earlier
and rely on that to protect obj->active_count.

Fixes: c5c1643c ("drm/msm: Drop struct_mutex from the retire path")
Signed-off-by: NRob Clark <robdclark@chromium.org>

Register GPU as a devfreq cooling device so that it can be passively
cooled by the thermal framework.
Signed-off-by: NAkhil P Oommen <akhilpo@codeaurora.org>
Tested-by: NMatthias Kaehlcke <mka@chromium.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Now that we are not relying on dev->struct_mutex to protect the
ring->submits lists, drop the struct_mutex lock.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NKristian H. Kristensen <hoegsberg@google.com>
Signed-off-by: NRob Clark <robdclark@chromium.org>

It cannot be atomically updated with obj->active_count, and the only
purpose is a useless WARN_ON() (which becomes a buggy WARN_ON() once
retire_submits() is not serialized with incoming submits via
struct_mutex)
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NKristian H. Kristensen <hoegsberg@google.com>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Before we remove dev->struct_mutex from the retire path, we have to deal
with the situation of a submit retiring before the submit ioctl returns.

To deal with this, ring->submits will hold a reference to the submit,
which is dropped when the submit is retired.  And the submit ioctl path
holds it's own ref, which it drops when it is done with the submit.

Also, add to submit list *after* getting/pinning bo's, to prevent badness
in case the completed fence is corrupted, and retire_worker mistakenly
believes the submit is done too early.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NKristian H. Kristensen <hoegsberg@google.com>
Signed-off-by: NRob Clark <robdclark@chromium.org>

One less place to rely on dev->struct_mutex.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NKristian H. Kristensen <hoegsberg@google.com>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Small cleanup, update_fences() is used in the hangcheck path, but also
in the normal retire path.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NKristian H. Kristensen <hoegsberg@google.com>
Signed-off-by: NRob Clark <robdclark@chromium.org>

It is somewhat redundant with the gpu tracepoints, and anyways not too
useful to justify spamming the log when debug traces are enabled.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NKristian H. Kristensen <hoegsberg@google.com>
Signed-off-by: NRob Clark <robdclark@chromium.org>

This also converts the special msm_gem_get_vaddr_active() to expect the
lock to already be held.  There are two call-sites for this, one already
has the lock held, so it is more straightforward to just open-code the
locking for the other caller.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NKristian H. Kristensen <hoegsberg@google.com>
Signed-off-by: NRob Clark <robdclark@chromium.org>

clk_prepare_enable() and clk_disable_unprepare() will check
NULL clock parameter, so It is not necessary to add additional checks.
Signed-off-by: NTian Tao <tiantao6@hisilicon.com>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Signed-off-by: NRob Clark <robdclark@chromium.org>

In the case where we have a back-to-back submission that shares the same
BO, this BO will be prematurely moved to inactive_list while retiring the
first submit. But it will be still part of the second submit which is
being processed by the GPU. Now, if the shrinker happens to be triggered at
this point, it will result in a premature purging of this BO.

To fix this, we need to refcount BO while doing submit and retire. Then,
it should be moved to inactive list when this refcount becomes 0.
Signed-off-by: NAkhil P Oommen <akhilpo@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

In $debugfs/gem we already show any vma(s) associated with an object.
Also show process names if the vma's address space is a per-process
address space.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NBjorn Andersson <bjorn.andersson@linaro.org>

Add support for allocating private address space instances. Targets that
support per-context pagetables should implement their own function to
allocate private address spaces.

The default will return a pointer to the global address space.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NBjorn Andersson <bjorn.andersson@linaro.org>

Now that we can get the ctx from the submitqueue, the extra arg is
redundant.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
[split out of previous patch to reduce churny noise]
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NBjorn Andersson <bjorn.andersson@linaro.org>

This will be populated by adreno-smmu, to provide a way for coordinating
enabling/disabling TTBR0 translation.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NBjorn Andersson <bjorn.andersson@linaro.org>

In a later patch, the drvdata will not directly be 'struct msm_gpu *',
so add a helper to reduce the churn.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NBjorn Andersson <bjorn.andersson@linaro.org>

Signed-off-by: NRob Clark <robdclark@chromium.org>