The docs say this is required for Gen7, and since the bit was added for
Gen6, we are also setting it there pit pf paranoia. Particularly as
Chris points out, if PIPE_CONTROL counts as a 3d state packet.

This was found through doc inspection by Ken and applies to Gen6+;
Reported-by: NKenneth Graunke <kenneth@whitecape.org>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NKeith Packard <keithp@keithp.com>

v2 by danvet: Use a new flag to flush the render target cache on gen6+
(hw reuses the old write flush bit), as suggested by Ben Widawsdy.
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NKenneth Graunke <kenneth@whitecape.org>
[danvet: this seems to fix cairo-perf-trace hangs on my snb]
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NKeith Packard <keithp@keithp.com>

"STALL_AT_SCOREBOARD" is much clearer than "STALL_EN" now that there are
several different kinds of stalls.  Also, "INSTRUCTION_CACHE_INVALIDATE"
is a lot easier to understand at a glance than the terse "IS_FLUSH."
Signed-off-by: NKenneth Graunke <kenneth@whitecape.org>
[danvet: use INVALIDATE for ro cache flags for more consistency]
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NKeith Packard <keithp@keithp.com>

Not all PIPE_CONTROLs have a length of 2, so remove it from the #define
and make each invocation specify the desired length.
Signed-off-by: NKenneth Graunke <kenneth@whitecape.org>
[danvet: implement style suggestion from Ben Widawsdy]
Signed-Off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NKeith Packard <keithp@keithp.com>

While I think the previous code is correct, it was hard to follow and
hard to debug. Since we already have a ring abstraction, might as well
use it to handle the semaphore updates and compares.

I don't expect this code to make semaphores better or worse, but you
never know...

v2:
Remove magic per Keith's suggestions.
Ran Daniel's gem_ring_sync_loop test on this.

v3:
Ignored one of Keith's suggestions.

v4:
Removed some bloat per Daniel's recommendation.

Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Keith Packard <keithp@keithp.com>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NKeith Packard <keithp@keithp.com>

Various issues involved with the space character were generating
warnings in the checkpatch.pl file. This patch removes most of those
warnings.
Signed-off-by: NAkshay Joshi <me@akshayjoshi.com>
Signed-off-by: NKeith Packard <keithp@keithp.com>

Prior to Ivybridge, the GFX_MODE would default to 0x800, meaning that
MI_FLUSH would flush the TLBs in addition to the rest of the caches
indicated in the MI_FLUSH command.  However starting with Ivybridge, the
register defaults to 0x2800 out of reset, meaning that to invalidate the
TLB we need to use PIPE_CONTROL.  Since we're not doing that yet, go
back to the old default so things work.

v2: don't forget to actually *clear* the new bit
Reviewed-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Tested-by: NKenneth Graunke <kenneth@whitecape.org>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Physically-addressed hardware status pages are initialized early in
the driver load process by i915_init_phys_hws. For UMS environments,
the ring structure is not initialized until the X server starts. At
that point, the entire ring structure is re-initialized with all new
values. Any values set in the ring structure (including
ring->status_page.page_addr) will be lost when the ring is
re-initialized.

This patch moves the initialization of the status_page.page_addr value
to intel_render_ring_init_dri.
Signed-off-by: NKeith Packard <keithp@keithp.com>
Cc: stable@kernel.org

[anholt v2: Don't forget that when going from cached to uncached, we
haven't been tracking the write domain from the CPU perspective, since
we haven't needed it for GPU coherency.]

[ickle v3: We also need to make sure we relinquish any fences on older
chipsets and clear the GTT for sane domain tracking.]
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This patch depends on patch "drm/i915: fix user irq miss in BSD ring on
g4x".
Once the previous patch apply, ring_get_irq/ring_put_irq become unused.
So simply remove them.
Signed-off-by: NFeng, Boqun <boqun.feng@intel.com>
Reviewed-by: NXiang, Haihao <haihao.xiang@intel.com>
Signed-off-by: NKeith Packard <keithp@keithp.com>

On g4x, user interrupt in BSD ring is missed.
This is because though g4x and ironlake share the same bsd_ring,
their interrupt control interfaces have _two_ differences.

1.different irq enable/disable functions:
On g4x are i915_enable_irq and i915_disable_irq.
On ironlake are ironlake_enable_irq and ironlake_disable_irq.
2.different irq flag:
On g4x user interrupt flag in BSD ring on is I915_BSD_USER_INTERRUPT.
On ironlake is GT_BSD_USER_INTERRUPT

Old bsd_ring_get/put_irq call ring_get_irq and ring_get_irq.
ring_get_irq and ring_put_irq only call ironlake_enable/disable_irq.
So comes the irq miss on g4x.

To fix this, as other rings' code do, conditionally call different
functions(i915_enable/disable_irq and ironlake_enable/disable_irq)
and use different interrupt flags in bsd_ring_get/put_irq.
Signed-off-by: NFeng, Boqun <boqun.feng@intel.com>
Reviewed-by: NXiang, Haihao <haihao.xiang@intel.com>
Cc: stable@kernel.org
Signed-off-by: NKeith Packard <keithp@keithp.com>

They have been moved from the ringbuffer groups to their own group it
looks like.  Fixes GPU hangs on gnome startup.
Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NKeith Packard <keithp@keithp.com>

Use Sandy Bridge paths in a few places.
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Reviewed-by: NKeith Packard <keithp@keithp.com>
Signed-off-by: NKeith Packard <keithp@keithp.com>

... to clarify just how we use it inside the driver and remove the
confusion of the poorly matching agp_type names. We still need to
translate through agp_type for interface into the fake AGP driver.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NKeith Packard <keithp@keithp.com>

Added a new function which waits for the ringbuffer space to be equal to
(total - 8). This is the empty condition of the ringbuffer, and
equivalent to head==tail.

Also modified two users of this functionality elsewhere in the code.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Required so that we don't obliterate the queue if initialising the
rings after the global IRQ handler is installed.

[Jesse, you recently looked at refactoring the IRQ installation
routines, does moving the initialisation of ring buffer data structures away
from that routine make sense in your grand scheme?]
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Reviewed-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NKeith Packard <keithp@keithp.com>

We always skipped flushing the BLT ring if the request flush did not
include the RENDER domain. However, this neglects that we try to flush
the COMMAND domain after every batch and before the breadcrumb interrupt
(to make sure the batch is indeed completed prior to the interrupt
firing and so insuring CPU coherency). As a result of the missing flush,
incoherency did indeed creep in, most notable when using lots of command
buffers and so potentially rewritting an active command buffer (i.e.
the GPU was still executing from it even though the following interrupt
had already fired and the request/buffer retired).

As all ring->flush routines now have the same preconditions, de-duplicate
and move those checks up into i915_gem_flush_ring().

Fixes gem_linear_blit.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=35284Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Tested-by: mengmeng.meng@intel.com

A lot of minor tweaks to fix the tracepoints, improve the outputting for
ftrace, and to generally make the tracepoints useful again. It is a start
and enough to begin identifying performance issues and gaps in our
coverage.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org

Instead of reporting EIO upfront in the entrance of an ioctl that may or
may not attempt to use the GPU, defer the actual detection of an invalid
ioctl to when we issue a GPU instruction. This allows us to continue to
use bo in video memory (via pread/pwrite and mmap) after the GPU has hung.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

During system suspend, the "wait for ring buffer to empty" loop would
always time out after three seconds, because the faster cached ring
buffer head read would always return zero.  Force the slow-and-careful
PIO read on all but the first iterations of the loop to fix it.

This also removes the unused (and useless) 'actual_head' variable that
tried to approximate doing this, but did it incorrectly.

Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Rafael J. Wysocki <rjw@sisk.pl>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: Dave Airlie <airlied@linux.ie>
Cc: DRI mailing list <dri-devel@lists.freedesktop.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

During suspend, Linus found that his machine would hang for 3 seconds,
and identified that intel_ring_buffer_wait() was the culprit:

"Because from looking at the code, I get the notion that
"intel_read_status_page()" may not be exact. But what happens if that
inexact value matches our cached ring->actual_head, so we never even
try to read the exact case? Does it _stay_ inexact for arbitrarily
long times? If so, we might wait for the ring to empty forever (well,
until the timeout - the behavior I see), even though the ring really
_is_ empty."

As the reported HEAD position is only updated every time it crosses a
64k boundary, whilst draining the ring it is indeed likely to remain one
value. If that value matches the last known HEAD position, we never read
the true value from the register and so trigger a timeout.
Reported-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

This is better handled through the tracepoints and just clutters the
debug logs.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

We weren't setting up the vfunc table when initialising the old DRI
ringbuffer, leading to such OOPSes as:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<(null)>] (null)
PGD 10c441067 PUD 1185e5067 PMD 0
Oops: 0010 [#1] PREEMPT SMP
last sysfs file: /sys/class/dmi/id/chassis_asset_tag
CPU 3
Modules linked in: i915 drm_kms_helper drm fb fbdev i2c_algo_bit
cfbcopyarea video backlight output cfbimgblt cfbfillrect autofs4 ipv6
nfs lockd fscache nfs_acl auth_rpcgss sunrpc coretemp hwmon_vid mousedev
usbhid hid option usb_wwan snd_hda_codec_via asus_atk0110 atl1e
usbserial snd_hda_intel snd_hda_codec firmware_class snd_hwdep snd_pcm
snd_seq snd_timer snd_seq_device processor parport_pc thermal snd
thermal_sys parport 8250_pnp button rng_core rtc_cmos shpchp hwmon
rtc_core ehci_hcd pci_hotplug uhci_hcd soundcore tpm_tis i2c_i801
rtc_lib tpm serio_raw snd_page_alloc tpm_bios i2c_core usbcore psmouse
intel_agp sg pcspkr sr_mod evdev cdrom ext3 jbd mbcache dm_mod sd_mod
ata_piix libata scsi_mod unix
Jan 18 15:49:29 lithui kernel:
Pid: 3605, comm: Xorg Not tainted 2.6.36.2 #5 P5KPL-CM/System Product
Name
RIP: 0010:[<0000000000000000>]  [<(null)>] (null)
RSP: 0018:ffff8801150d1d40  EFLAGS: 00010202
RAX: 000000000001ffff RBX: ffff88011a011b00 RCX: 000000000001a704
RDX: ffff880118566028 RSI: ffff880118566028 RDI: ffff880117876800
RBP: ffff8801150d1d48 R08: ffff8801195fe300 R09: 00000000c0086444
R10: 0000000000000001 R11: 0000000000003206 R12: ffff880117876800
R13: ffff880118566000 R14: ffff880117876820 R15: ffff8801150d1df8
FS:  00007f1038d456e0(0000) GS:ffff880001780000(0000)
knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000001187e7000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process Xorg (pid: 3605, threadinfo ffff8801150d0000, task
ffff88011b016e40)
Stack:
ffffffffa043b8e6 ffff8801150d1d98 ffffffffa041768b dead000000000000
<0> 0000000000000048 00007f1023f2a000 0000000000000044 0000000000000008
<0> ffff88010d26bd80 ffff880117876800 ffff8801150d1df8 ffff8801150d1ea8
Call Trace:
[<ffffffffa043b8e6>] ? intel_ring_advance+0x16/0x20 [i915]
[<ffffffffa041768b>] i915_irq_emit+0x15b/0x240 [i915]
[<ffffffffa03ea7b1>] drm_ioctl+0x1f1/0x460 [drm]
[<ffffffffa0417530>] ? i915_irq_emit+0x0/0x240 [i915]
[<ffffffff810dd8f1>] ? do_sync_read+0xd1/0x120
[<ffffffff81025b1f>] ? do_page_fault+0x1df/0x3d0
[<ffffffff810ed5c7>] do_vfs_ioctl+0x97/0x550
[<ffffffff8115c2ea>] ? security_file_permission+0x7a/0x90
[<ffffffff810edb19>] sys_ioctl+0x99/0xa0
[<ffffffff810024ab>] system_call_fastpath+0x16/0x1b
Code:  Bad RIP value.
RIP  [<(null)>] (null)
RSP <ffff8801150d1d40>
CR2: 0000000000000000
Reported-by: NHerbert Xu <herbert@gondor.apana.org.au>
Tested-by: NHerbert Xu <herbert@gondor.apana.org.au>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=29153
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=23172Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org

As the IMR for the USER interrupts are not modified elsewhere, we can
separate the spinlock used for these from that of hpd and pipestats.
Those two IMR are manipulated under an IRQ and so need heavier locking.
Reported-and-tested-by: NAlexey Fisher <bug-track@fisher-privat.net>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

... and move it under the spinlock to gain the appropriate memory
barriers.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=32752Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Otherwise we may consume 20% of the CPU just handling IRQs whilst
rendering. Ouch.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

On i830 if the tail pointer is set to within 2 cachelines of the end of
the buffer, the chip may hang. So instead if the tail were to land in
that location, we pad the end of the buffer with NOPs, and start again
at the beginning.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Restore PIPE_CONTROL once again just for Ironlake, as it appears that
MI_USER_INTERRUPT does not have the same coherency guarantees, that is
on Ironlake the interrupt following a GPU write is not guaranteed to
arrive after the write is coherent from the CPU, as it does on the
other generations.
Reported-by: NZhenyu Wang <zhenyuw@linux.intel.com>
Reported-by: NShuang He <shuang.he@intel.com>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=32402Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

In order to enforce the correct memory barriers for irq get/put, we need
to perform the actual counting using atomic operations.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

If the tail advances beyond the autoreport HEAD value, then we need to
fallback to an uncached read of the HEAD register in order to ascertain
the correct amount of remaining space in the ringbuffer.
Reported-by: NFang, Xun <xunx.fang@intel.com>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=32259Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

The workaround is hideous and we are using the STORE_DWORD on all other
generations on all other rings, so use for the gen5 render ring as
well.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

There's not much we can do here but hope for the best. However the first
failure happens quite frequently and if often remedied by the second
attempt to reset HEAD. So only print the error if that attempt also
fails.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=19802Reported-by: NThomas Meyer <thomas@m3y3r.de>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: stable@kernel.org

The bulk of the change is to convert the growing list of rings into an
array so that the relationship between the rings and the semaphore sync
registers can be easily computed.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

This makes the various rings more consistent by removing the anomalous
handing of the rendering ring execbuffer dispatch.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

The pipe control object is allocated by the device for the sole use of the
render ringbuffer. Move this detail from the general code to the render
ring buffer initialisation.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

A glorified s/obj_priv/obj/ with a net reduction of over a 100 lines and
many characters!
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>