Before adding another lock, give ring->lock a more descriptive name.
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NKristian H. Kristensen <hoegsberg@google.com>
Signed-off-by: NRob Clark <robdclark@chromium.org>

The microcode bo's should never be madvise(WONTNEED), so these should
not be using msm_gem_get_vaddr_active().
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NKristian H. Kristensen <hoegsberg@google.com>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Signed-off-by: NRob Clark <robdclark@chromium.org>

Commit 604234f3 ("drm/msm: Enable expanded apriv support for a650")
was checking the result of adreno_is_a650() before the gpu revision
got probed in adreno_gpu_init() so it was always coming across as
false. Snoop into the revision ID ahead of time to correctly set the
hw_apriv flag so that it can be used by msm_gpu to properly setup
global buffers.

Fixes: 604234f3 ("drm/msm: Enable expanded apriv support for a650")
Reported-by: NJonathan Marek <jonathan@marek.ca>
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Tested-by: NJonathan Marek <jonathan@marek.ca>
Signed-off-by: NRob Clark <robdclark@chromium.org>

As newer GPU families are added it makes less sense to maintain a
"generic" version functions for older families. Move adreno_submit()
and get_rptr() into the target specific code for a2xx, a3xx and a4xx.
Add a parameter to adreno_flush to pass the target specific WPTR register
instead of relying on the generic register.

All of this gets rid of the last of the REG_ADRENO offsets so remove all
all the register definitions and infrastructure.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Support the WHERE_AM_I opcode for the A618, A630 and A640 GPUs if the
microcode supports it. The WHERE_AM_I opcode allows the RPTR shadow
to be updated in priviliged memory which protects the shadow from being
read or written from user submissions.

A650 already supports extended APRIV have built in hardware support for
to access privilged memory from the CP and can go back to using the
hardware RPTR shadow feature.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Add support for using per-instance pagetables if all the dependencies are
available.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NAkhil P Oommen <akhilpo@codeaurora.org>

Now that we can get the ctx from the submitqueue, the extra arg is
redundant.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
[split out of previous patch to reduce churny noise]
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NBjorn Andersson <bjorn.andersson@linaro.org>

Signed-off-by: NRob Clark <robdclark@chromium.org>

Disable the RPTR shadow across all targets. It will be selectively
re-enabled later for targets that need it.

Cc: stable@vger.kernel.org
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

a650 supports expanded apriv support that allows us to map critical buffers
(ringbuffer and memstore) as as privileged to protect them from corruption.

Cc: stable@vger.kernel.org
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Replace the existing /* fall through */ comments and its variants with
the new pseudo-keyword macro fallthrough[1]. Also, remove unnecessary
fall-through markings when it is the case.

[1] https://www.kernel.org/doc/html/v5.7/process/deprecated.html?highlight=fallthrough#implicit-switch-case-fall-throughSigned-off-by: NGustavo A. R. Silva <gustavoars@kernel.org>

Initialize hardware clock-gating registers on A640 and A650 GPUs.

At least for A650, this solves some performance issues.
Signed-off-by: NJonathan Marek <jonathan@marek.ca>
Signed-off-by: NRob Clark <robdclark@chromium.org>

This will allow supporting different hwcg tables for a6xx.
Signed-off-by: NJonathan Marek <jonathan@marek.ca>
Signed-off-by: NRob Clark <robdclark@chromium.org>

This is required for A640 and A650 to be able to share UBWC-compressed
images with other HW such as display, which expect this configuration.
Signed-off-by: NJonathan Marek <jonathan@marek.ca>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Small cleanup, lets not open-code bits/bitfields that are properly
defined in the rnndb xml (and therefore have builders in the generated
headers)
Signed-off-by: NRob Clark <robdclark@chromium.org>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

We don't want it under CONFIG_DRM_MSM_GPU_STATE, we need it all the
time (like the other GPUs do).

Fixes: ccac7ce3 ("drm/msm: Refactor address space initialization")
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Refactor how address space initialization works. Instead of having the
address space function create the MMU object (and thus require separate but
equal functions for gpummu and iommu) use a single function and pass the
MMU struct in. Make the generic code cleaner by using target specific
functions to create the address space so a2xx can do its own thing in its
own space.  For all the other targets use a generic helper to initialize
IOMMU but leave the door open for newer targets to use customization
if they need it.
Reviewed-by: NRob Clark <robdclark@gmail.com>
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Tested-by: NShawn Guo <shawn.guo@linaro.org>
[squash in rebase fixups]
Signed-off-by: NRob Clark <robdclark@chromium.org>

Spelling out _unlocked for each and every driver is a annoying.
Especially if we consider how many drivers, do not know (or need to)
about the horror stories involving struct_mutex.

Just drop the suffix. It makes the API cleaner.

Done via the following script:

__from=drm_gem_object_put_unlocked
__to=drm_gem_object_put
for __file in $(git grep --name-only $__from); do
  sed -i  "s/$__from/$__to/g" $__file;
done

Cc: Rob Clark <robdclark@gmail.com>
Cc: Sean Paul <sean@poorly.run>
Cc: David Airlie <airlied@linux.ie>
Signed-off-by: NEmil Velikov <emil.velikov@collabora.com>
Acked-by: NSam Ravnborg <sam@ravnborg.org>
Acked-by: NThomas Zimmermann <tzimmermann@suse.de>
Link: https://patchwork.freedesktop.org/patch/msgid/20200515095118.2743122-25-emil.l.velikov@gmail.com

Adreno 640 and 650 GPUs need some registers set differently.
Signed-off-by: NJonathan Marek <jonathan@marek.ca>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Add HFI v2 code paths required by Adreno 640 and 650 GPUs.
Signed-off-by: NJonathan Marek <jonathan@marek.ca>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Writing to the devfreq sysfs nodes while the GPU is powered down can
result in a system crash (on a5xx) or a nasty GMU error (on a6xx):

 $ /sys/class/devfreq/5000000.gpu# echo 500000000 > min_freq
  [  104.841625] platform 506a000.gmu: [drm:a6xx_gmu_set_oob]
	*ERROR* Timeout waiting for GMU OOB set GPU_DCVS: 0x0

Despite the fact that we carefully try to suspend the devfreq device when
the hardware is powered down there are lots of holes in the governors that
don't check for the suspend state and blindly call into the devfreq
callbacks that end up triggering hardware reads in the GPU driver.

Call pm_runtime_get_if_in_use() in the gpu_busy() and gpu_set_freq()
callbacks to skip the hardware access if it isn't active.

v3: Only check pm_runtime_get_if_in_use() for == 0 per Eric Anholt
v2: Use pm_runtime_get_if_in_use() per Eric Anholt

Cc: stable@vger.kernel.org
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Commit e812744c ("drm: msm: a6xx: Add support for A618") missed
updating the VBIF flush in a6xx_gmu_shutdown and instead
inserted the new sequence into a6xx_pm_suspend along with a redundant
GMU idle.

Move a6xx_bus_clear_pending_transactions to a6xx_gmu.c and use it in
the appropriate place in the shutdown routine and remove the redundant
idle call.

v2: Remove newly unused variable that was triggering a warning
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NRob Clark <robdclark@gmail.com>
Fixes: e812744c ("drm: msm: a6xx: Add support for A618")
Tested-by: NDouglas Anderson <dianders@chromium.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Commit e812744c ("drm: msm: a6xx: Add support for A618") added a
universal GBIF un-halt into a6xx_start(). This can cause problems for
a630 targets which do not use GBIF and might have access protection
enabled on the region now occupied by the GBIF registers.

But it turns out that we didn't need to unhalt the GBIF in this path
since the stop function already takes care of that after executing a flush
but before turning off the headswitch. We should be confident that the
GBIF is open for business when we restart the hardware.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Tested-by: NJohn Stultz <john.stultz@linaro.org>
Reviewed-by: NRob Clark <robdclark@gmail.com>
Fixes: e812744c ("drm: msm: a6xx: Add support for A618")
Signed-off-by: NRob Clark <robdclark@chromium.org>

Highest bank bit configuration is different for a618 gpu. Update
it with the correct configuration which is the reset value incidentally.
Signed-off-by: NAkhil P Oommen <akhilpo@codeaurora.org>
Signed-off-by: NSharat Masetty <smasetty@codeaurora.org>
Fixes: e812744c ("drm: msm: a6xx: Add support for A618")
Reviewed-by: NRob Clark <robdclark@gmail.com>
Signed-off-by: NRob Clark <robdclark@chromium.org>

This patch adds support for enabling Graphics Bus Interface(GBIF)
used in multiple A6xx series chipets. Also makes changes to the
PDC/RSC sequencing specifically required for A618. This is needed
for proper interfacing with RPMH.
Signed-off-by: NSharat Masetty <smasetty@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

We can have two cases, when it comes to "zap" fw.  Either the fw
requires zap fw to take the GPU out of secure mode at boot, or it does
not and we can write RBBM_SECVID_TRUST_CNTL directly.  Previously we
decided based on whether zap fw load succeeded, but this is not a great
plan because:

1) we could have zap fw in the filesystem on a device where it is not
   required
2) we could have the inverse case

Instead, shift to deciding based on whether we have a 'zap-shader' node
in dt.  In practice, there is only one device (currently) with upstream
dt that does not use zap (cheza), and it already has a /delete-node/ for
the zap-shader node.

Fixes: abccb9fe ("drm/msm/a6xx: Add zap shader load")
Signed-off-by: NRob Clark <robdclark@chromium.org>

Explicitly mark intentional fall throughs in switch statements to keep
-Wimplicit-fallthrough from complaining.
Reviewed-by: NRob Clark <robdclark@gmail.com>
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NSean Paul <seanpaul@chromium.org>
Link: https://patchwork.freedesktop.org/patch/msgid/1564073588-27386-1-git-send-email-jcrouse@codeaurora.org

A5XX and newer GPUs can be run in either 32 or 64 bit mode. The GPU
registers and the microcode use 64 bit virtual addressing in either
case but the upper 32 bits are ignored if the GPU is in 32 bit mode.
There is no performance disadvantage to remaining in 64 bit mode even
if we are only generating 32 bit addresses so switch over now to prepare
for using addresses above 4G on targets that support them.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

This rename makes it more clear that everything initialized in the _init
function must be cleaned up in a6xx_gmu_remove. This will hopefully
dissuade people from using device managed resources (for reasons laid
out in the previous patch).

Changes in v2:
- None

Cc: Jordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NSean Paul <seanpaul@chromium.org>
Link: https://patchwork.freedesktop.org/patch/msgid/20190523171653.138678-6-sean@poorly.run

Depending on platform firmware, a zap shader may not be required to take
the GPU out of secure mode on boot, in which case we can just write
RBBM_SECVID_TRUST_CNTL directly.  Which we *mostly* handled, but missed
clearing 'ret' resulting that hw_init() returned an error on these
devices.

Fixes: abccb9fe ("drm/msm/a6xx: Add zap shader load")
Acked-by: NJordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: NSean Paul <sean@poorly.run>
Signed-off-by: NRob Clark <robdclark@chromium.org>
Signed-off-by: NSean Paul <seanpaul@chromium.org>
Link: https://patchwork.freedesktop.org/patch/msgid/20190508130726.27557-1-robdclark@gmail.com

Add CONFIG_DRM_MSM_GPU_STATE to conditionally compile Adreno GPU state
code depending on the availability of the dependencies.
Reported-by: NHulk Robot <hulkci@huawei.com>
Reported-by: NYueHaibing <yuehaibing@huawei.com>
Fixes: 1707add8 ("drm/msm/a6xx: Add a6xx gpu state")
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

The a6xx GPU powers on in secure mode which restricts what memory it can
write to. To get out of secure mode the GPU driver can write to
REG_A6XX_RBBM_SECVID_TRUST_CNTL but on targets that are "secure" that
register region is blocked and writes will cause the system to go down.

For those targets we need to execute a special sequence that involves
loadinga special shader that clears the GPU registers and use a PM4
sequence to pull the GPU out of secure. Add support for loading the zap
shader and executing the secure sequence. For targets that do not support
SCM or the specific SCM sequence this should fail and we would fall back
to writing the register.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Currently if the GMU resume function fails all we try to do is clear the
BOOT_SLUMBER oob which usually times out and ends up in a cycle of death.
If the resume function fails at any point remove any RPMh votes that might
have been added and try to shut down the GMU hardware cleanly.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

Now that the GX domain is sorted we can wire up a working GMU reset.
IF a GMU hang was detected then try to forcefully shut down the GMU
in the power down sequence which should ensure that it can recover
normally on the next power up.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@chromium.org>

For debugging purposes it is useful to assign descriptions
to buffers so that we know what they are used for. Add
a field to the buffer object and use that to name the various
kernel side allocations which ends up looking like like this
in /d/dri/X/gem:

   flags       id ref  offset   kaddr            size     madv      name
   00040000: I  0 ( 1) 00000000 0000000070b79eca 00004096           memptrs
      vmas: [gpu: 01000000,mapped,inuse=1]
   00020000: I  0 ( 1) 00000000 0000000031ed4074 00032768           ring0
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@gmail.com>

Add a reference count to track how many times a particular
chunk of iova memory is pinned (mapped) in the iomu and
add msm_gem_unpin_iova to give up references.

It is important to note that msm_gem_unpin_iova replaces
msm_gem_put_iova because the new implicit behavior
that an assigned iova in a given vma is now valid for the
life of the buffer and what we are really focusing on is
the use of that iova.

For now the unmappings are lazy; once the reference counts
go to zero they *COULD* be unmapped dynamically but that
will require an outside force such as a shrinker or
mm_notifiers.  For now, we're just focusing on getting
the counting right and setting ourselves up to be ready
for the future.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@gmail.com>

Add support for gathering and dumping the a6xx GPU state including
registers, GMU registers, indexed registers, shader blocks,
context clusters and debugbus.

v2: Fix bugs discovered by Sharat Masetty
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@gmail.com>

Add trace events to track the progress of a GPU submission
msm_gpu_submit occurs at the beginning of the submissions,
msm_gpu_submit_flush happens when the submission is put on
the ringbuffer and msm_submit_flush_retired is sent when
the operation is retired.

To make it easier to track the operations a unique sequence
number is assigned to each submission and displayed in each
event output so a human or a script can easily associate
the events related to a specific submission.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@gmail.com>

Add infrastructure to track statistics for GPU submissions
by sampling certain perfcounters before and after a submission.

To store the statistics, the per-ring memptrs region is
expanded to include room for up to 64 entries - this should
cover a reasonable amount of inflight submissions without
worrying about losing data. The target specific code inserts
PM4 commands to sample the counters before and after
submission and store them in the data region. The CPU can
access the data after the submission retires to make sense
of the statistics and communicate them to the user.
Signed-off-by: NJordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: NRob Clark <robdclark@gmail.com>