Previously the queue/pipe split with kfd operated with pipe
granularity. This patch allows amdgpu to take ownership of an arbitrary
set of queues.

It also consolidates the last few magic numbers in the compute
initialization process into mec_init.

v2: support for gfx9
v3: renamed AMDGPU_MAX_QUEUES to AMDGPU_MAX_COMPUTE_QUEUES
v4: fix off-by-one in num_mec checks in *_compute_queue_acquire
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Make amdgpu the owner of all per-pipe state of the HQDs.

This change will allow us to split the queues between kfd and amdgpu
with a queue granularity instead of pipe granularity.

This patch fixes kfd allocating an HDP_EOP region for its 3 pipes which
goes unused.

v2: support for gfx9
v3: fix gfx7 HPD intitialization
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Take ownership of pipe initialization away from KFD.

Note that hpd_eop_gpu_addr was already large enough to accomodate all
pipes.
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Rename straggler instances of r(adeon)dev to a(mdgpu)dev
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The return value from copy_form_user is 0 for the success case.
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Use the same gfx_*_mqd_commit function for kfd and amdgpu codepaths.

This removes the last duplicates of this programming sequence.

v2: fix cp_hqd_pq_wptr value
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The gfxv7 contains a slightly different version of cik_mqd called
bonaire_mqd. This can introduce subtle bugs if fixes are not applied in
both places.
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Acked-by: NChristian König <christian.koenig@amd.com>
Acked-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Handle HQD deactivation timeouts instead of ignoring them.
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Acked-by: NChristian König <christian.koenig@amd.com>
Acked-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The MQD programming sequence currently exists in 3 different places.
Refactor it to absorb all the duplicates.

The success path remains mostly identical except for a slightly
different order in the non-kiq case. This shouldn't matter if the HQD
is disabled.

The error handling paths have been updated to deal with the new code
structure.

v2: the non-kiq path for gfxv8 was dropped in the rebase
v3: split MEC_HPD_SIZE rename, dropped doorbell changes
Reviewed-by: NEdward O'Callaghan <funfunctor@folklore1984.net>
Acked-by: NChristian König <christian.koenig@amd.com>
Acked-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Rename MEC_HPD_SIZE to GFXN_MEC_HPD_SIZE to clarify it is specific to a
gfx generation.
Signed-off-by: NAndres Rodriguez <andresx7@gmail.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This reverts commit f8fdaa0e7b81698ba2ad8c2d20c7f9a44c75e0c6.
firmware add support for this feature, so still ctrl by vbios.
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This isn't beneficial any more since VRAM allocations are now split
so that they fits into a single page table.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Makes it easier to update the PDE with huge pages.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

 Make the code easier to understand.
Signed-off-by: NAlex Xie <AlexBin.Xie@amd.com>
Reviewed-by: NMichel Dänzer <michel.daenzer@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

  Move several if statements and a loop statment from
  run time to initialization time.
Signed-off-by: NAlex Xie <AlexBin.Xie@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

We need program ring buffer on instance 1 register space domain,
when only if instance 1 available, with two instances or instance 0,
and we need only program instance 0 regsiter space domain for ring.
Signed-off-by: NLeo Liu <leo.liu@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

This change is also useful for the upcoming changes where page tables
can be updated by CPU.
Signed-off-by: NHarish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Reviewed-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

If updating the PDs fails we now invalidate all entries to try again later.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NChunming Zhou <david1.zhou@amd.com>
Reviewed-by: NJunwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Rename adjust_mc_addr to get_vm_pde and check the address bits in one place.

v2: handle vcn as well, keep setting the valid bit manually,
    add a BUG_ON() for GMC v6, v7 and v8 as well.
v3: handle vcn_v1_0_enc_ring_emit_vm_flush as well.
v4: fix the BUG_ON mask for GFX6-8
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NFelix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Load Balancing Per Watt (LBPW) allows dynamically disable CUs
when they are idle
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

remnants from bring-up.
Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Tools fb address was failed to send to smu when smu
was not running. Changing sequence will fix it.
Signed-off-by: NEric Huang <JinHuiEric.Huang@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

It is to fix bug of sysfs entry pp_table which had size 0 of output before.
Signed-off-by: NEric Huang <JinHuiEric.Huang@amd.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

amdgpu_device_resume() & amdgpu_device_init() have a high
time consuming call of amdgpu_late_init() which sets the
clock_gating state of all IP blocks and is blocking.
This patch defers only this setting of clock gating state
operation to post resume of amdgpu driver but ideally before
the UI comes up or in some cases post ui as well.

With this change the resume time of amdgpu_device comes down
from 1.299s to 0.199s which further helps in reducing the overall
system resume time.

V1: made the optimization applicable during driver load as well.

TEST:(For ChromiumOS on STONEY only)
* UI comes up
* amdgpu_late_init() call gets called consistently and no errors reported.
Signed-off-by: NShirish S <shirish.s@amd.com>
Reviewed-by: NHuang Rui <ray.huang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

this bug happened when amdgpu load failed.

[   75.740951] BUG: unable to handle kernel paging request at 00000000000031c0
[   75.748167] IP: [<ffffffffa064a0e0>] amdgpu_fbdev_restore_mode+0x20/0x60 [amdgpu]
[   75.755774] PGD 0

[   75.759185] Oops: 0000 [#1] SMP
[   75.762408] Modules linked in: amdgpu(OE-) ttm(OE) drm_kms_helper(OE) drm(OE) i2c_algo_bit(E) fb_sys_fops(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) rpcsec_gss_krb5(E) nfsv4(E) nfs(E) fscache(E) eeepc_wmi(E) asus_wmi(E) sparse_keymap(E) intel_rapl(E) snd_hda_codec_hdmi(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) snd_hda_intel(E) snd_hda_codec(E) snd_hda_core(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) snd_hwdep(E) snd_pcm(E) snd_seq_midi(E) coretemp(E) kvm_intel(E) snd_seq_midi_event(E) snd_rawmidi(E) kvm(E) snd_seq(E) joydev(E) snd_seq_device(E) snd_timer(E) irqbypass(E) crct10dif_pclmul(E) crc32_pclmul(E) mei_me(E) ghash_clmulni_intel(E) snd(E) aesni_intel(E) mei(E) soundcore(E) aes_x86_64(E) shpchp(E) serio_raw(E) lrw(E) acpi_pad(E) gf128mul(E) glue_helper(E) ablk_helper(E) mac_hid(E)
[   75.835574]  cryptd(E) parport_pc(E) ppdev(E) lp(E) nfsd(E) parport(E) auth_rpcgss(E) nfs_acl(E) lockd(E) grace(E) sunrpc(E) autofs4(E) hid_generic(E) usbhid(E) mxm_wmi(E) psmouse(E) e1000e(E) ptp(E) pps_core(E) ahci(E) libahci(E) wmi(E) video(E) i2c_hid(E) hid(E)
[   75.858489] CPU: 5 PID: 1603 Comm: rmmod Tainted: G           OE   4.9.0-custom #2
[   75.866183] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[   75.875050] task: ffff88045d1bbb80 task.stack: ffffc90002de4000
[   75.881094] RIP: 0010:[<ffffffffa064a0e0>]  [<ffffffffa064a0e0>] amdgpu_fbdev_restore_mode+0x20/0x60 [amdgpu]
[   75.891238] RSP: 0018:ffffc90002de7d48  EFLAGS: 00010286
[   75.896648] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000001
[   75.903933] RDX: 0000000000000000 RSI: ffff88045d1bbb80 RDI: 0000000000000286
[   75.911183] RBP: ffffc90002de7d50 R08: 0000000000000502 R09: 0000000000000004
[   75.918449] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880464bf0000
[   75.925675] R13: ffffffffa0853000 R14: 0000000000000000 R15: 0000564e44f88210
[   75.932980] FS:  00007f13d5400700(0000) GS:ffff880476540000(0000) knlGS:0000000000000000
[   75.941238] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   75.947088] CR2: 00000000000031c0 CR3: 000000045fd0b000 CR4: 00000000003406e0
[   75.954332] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   75.961566] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   75.968834] Stack:
[   75.970881]  ffff880464bf0000 ffffc90002de7d60 ffffffffa0636592 ffffc90002de7d80
[   75.978454]  ffffffffa059015f ffff880464bf0000 ffff880464bf0000 ffffc90002de7da8
[   75.986076]  ffffffffa0595216 ffff880464bf0000 ffff880460f4d000 ffffffffa0853000
[   75.993692] Call Trace:
[   75.996177]  [<ffffffffa0636592>] amdgpu_driver_lastclose_kms+0x12/0x20 [amdgpu]
[   76.003700]  [<ffffffffa059015f>] drm_lastclose+0x2f/0xd0 [drm]
[   76.009777]  [<ffffffffa0595216>] drm_dev_unregister+0x16/0xd0 [drm]
[   76.016255]  [<ffffffffa0595944>] drm_put_dev+0x34/0x70 [drm]
[   76.022139]  [<ffffffffa062f365>] amdgpu_pci_remove+0x15/0x20 [amdgpu]
[   76.028800]  [<ffffffff81416499>] pci_device_remove+0x39/0xc0
[   76.034661]  [<ffffffff81531caa>] __device_release_driver+0x9a/0x140
[   76.041121]  [<ffffffff81531e58>] driver_detach+0xb8/0xc0
[   76.046575]  [<ffffffff81530c95>] bus_remove_driver+0x55/0xd0
[   76.052401]  [<ffffffff815325fc>] driver_unregister+0x2c/0x50
[   76.058244]  [<ffffffff81416289>] pci_unregister_driver+0x29/0x90
[   76.064466]  [<ffffffffa0596c5e>] drm_pci_exit+0x9e/0xb0 [drm]
[   76.070507]  [<ffffffffa0796d71>] amdgpu_exit+0x1c/0x32 [amdgpu]
[   76.076609]  [<ffffffff81104810>] SyS_delete_module+0x1a0/0x200
[   76.082627]  [<ffffffff810e2b1a>] ? rcu_eqs_enter.isra.36+0x4a/0x50
[   76.089001]  [<ffffffff8100392e>] do_syscall_64+0x6e/0x180
[   76.094583]  [<ffffffff817e1d2f>] entry_SYSCALL64_slow_path+0x25/0x25
[   76.101114] Code: 94 c0 c3 31 c0 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 31 c0 48 89 e5 53 48 89 fb 48 c7 c7 1d 21 84 a0 e8 ab 77 b3 e0 e8 fc 8b d7 e0 <48> 8b bb c0 31 00 00 48 85 ff 74 09 e8 ff eb fc ff 85 c0 75 03
[   76.121432] RIP  [<ffffffffa064a0e0>] amdgpu_fbdev_restore_mode+0x20/0x60 [amdgpu]
Signed-off-by: NRex Zhu <Rex.Zhu@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Signed-off-by: NHawking Zhang <Hawking.Zhang@amd.com>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Same as the previous patch, but for pageflipping now. This also lets us
clear up the copy paste for vblank/vline IRQs.

Changes since v1:
- Preserve the order all registers are written back
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NLyude <lyude@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

Same as the previous patch, but now for handling HDMI audio interrupts.

Changes since v1:
- Preserve the order we write back all registers
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NLyude <lyude@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>

The current code here is really, really bad. A huge amount of it looks
to be copy pasted, it has some weird hatred of arrays and code sharing,
switch cases everywhere for things that really don't need them, and it
makes the file seem immensely more complex then it actually is. This is
a pain for maintanence, and is vulnerable to more weird irq handling
bugs.

So, let's start cleaning this up a bit. Modify all of the IRQ handlers
for evergreen/si so that they just use for loops. As well, we add a
helper function radeon_irq_kms_set_irq_n_enabled(), whose purpose is
just to update the state of registers that enable/disable interrupts
while printing any changes to the set of enabled interrupts to the
kernel log.

Note in this commit, since vblank/vline irq acking is intertwined with
page flip irq acking, we can't cut out all of the copy paste in
evergreen/si_irq_ack() just yet.

Changes since v1:
- Preserve order we write back all registers
Acked-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NLyude <lyude@redhat.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>