Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

As evict_something() is called by routines that do not repeatedly search
again, try harder in the initial search to find an object that matches
the request.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

First the routine attempted to unlock a mutex it did not own along the
error path.

Secondly the routine should never be called on any list but the inactive
one, since we attempt to unbind those objects, so fix the calling semantics.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

By adding tracepoint equivalents for WATCH_BUF/EXEC we are able to monitor
the lifetimes of objects, requests and significant events. These events can
then be probed using the tracing frameworks, such as systemtap and, in
particular, perf.

For example to record the stack trace for every GPU stall during a run, use

  $ perf record -e i915:i915_gem_request_wait_begin -c 1 -g

And

  $ perf report

to view the results.

[Updated to fix compilation issues caused.]
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Ben Gamari <bgamari@gmail.com>
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>

If the presumed_offset as feed to userspace and returned to the kernel
from a previous execbuffer is still valid, then we do not need to rewrite
the relocation entry and may skip the offset sanity checks.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Eric noted a potential concern with the low bits not being strictly used
as part of the absolute offset (instead part of the command stream to the
GPU), but in practice that should not be an issue.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Tested-by: NAndy Whitcroft <apw@canonical.com>
Cc: Eric Anholt <eric@anholt.net>
CC: stable@kernel.org
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Due to the necessity of having to take the struct_mutex, the i915
shrinker can not free the inactive lists if we fail to allocate memory
whilst processing a batch buffer, triggering an OOM and an ENOMEM that
is reported back to userspace. In order to fare better under such
circumstances we need to manually retry a failed allocation after
evicting inactive buffers.

To do so involves 3 steps:
1. Marking the backing shm pages as NORETRY.
2. Updating the get_pages() callers to evict something on failure and then
   retry.
3. Revamping the evict something logic to be smarter about the required
   buffer size and prefer to use volatile or clean inactive pages.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Similar to the madvise() concept, the application may wish to mark some
data as volatile. That is in the event of memory pressure the kernel is
free to discard such buffers safe in the knowledge that the application
can recreate them on demand, and is simply using these as a cache.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

This should help GEM handle memory pressure sitatuions more gracefully.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

There is no need to store the gtt_alignment as it is either explicitly
set according to the hardware requirements (e.g. scanout) or the
minimum alignment is computed on demand.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

If we failed to set the domain, the buffer was no longer being tracked
on any list.

Cc: stable@kernel.org
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

There is a very real possibility that multiple CPUs will notice that the
GPU is wedged. This introduces all sorts of potential race conditions.
Make the wedged flag atomic to mitigate this risk.
Signed-off-by: NBen Gamari <bgamari.foss@gmail.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

We set a periodic timer to check on the GPU, resetting it every time a
batch is completed. If the timer elapses, we check acthd. If acthd
hasn't changed in two timer periods, we assume the chip is wedged.

This is implemented in such a way that it leaves the option open to
employ adaptive timer intervals in the future. One could wait until
several timer periods have elapsed before declaring the chip dead. If
the chip comes back after several periods but before the "dead"
threshold, the timer interval or dead threshold could be raised.

It is important to note that while checking for active requests, we need
to account for the fact that requests are removed from the list (i.e.
retired) in a deferred work queue handler. This means that merely
checking for an empty request_list is insufficient; the list could be
non-empty yet the GPU still idle, causing the hangcheck timer to
incorrectly mark the GPU as wedged (it took me a while to figure that
out---sigh...)
Signed-off-by: NBen Gamari <bgamari.foss@gmail.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

We'll need it in i915_irq.c for checking whether there are outstanding
requests. Also, the function really ought to return a bool, not an int.
Signed-off-by: NBen Gamari <bgamari.foss@gmail.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

i915_wait_request() only checks mm.wedged after it interacts with the
hardware, generally causing the driver to lock up waiting for a wedged
chip. Make sure we check mm.wedged as the first thing we do.
Reported-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NBen Gamari <bgamari.foss@gmail.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>

drm_ht_remove_item() does not handle removing an absent item and the hlist
in particular is incorrectly initialised. The easy remedy is simply skip
calling i915_gem_free_mmap_offset() unless we have actually created the
offset and associated ht entry.

This also fixes the mishandling of a partially constructed offset which
leaves pointers initialized after freeing them along the
i915_gem_create_mmap_offset() error paths.

In particular this should fix the oops found here:
https://bugs.launchpad.net/ubuntu/+source/xserver-xorg-video-intel/+bug/415357/comments/8Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>
Cc: stable@kernel.org

Ever since we enabled GEM, the pre-9xx chipsets (particularly 865) have had
serious stability issues.  Back in May a wbinvd was added to the DRM to
work around much of the problem.  Some failure remained -- easily visible
by dragging a window around on an X -retro desktop, or by looking at bugzilla.

The chipset flush was on the right track -- hitting the right amount of
memory, and it appears to be the only way to flush on these chipsets, but the
flush page was mapped uncached.  As a result, the writes trying to clear the
writeback cache ended up bypassing the cache, and not flushing anything!  The
wbinvd would flush out other writeback data and often cause the data we wanted
to get flushed, but not always.  By removing the setting of the page to UC
and instead just clflushing the data we write to try to flush it, we get the
desired behavior with no wbinvd.

This exports clflush_cache_range(), which was laying around and happened to
basically match the code I was otherwise going to copy from the DRM.
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NBrice Goglin <Brice.Goglin@ens-lyon.org>
Cc: stable@kernel.org

Otherwise, some other userland writing into its buffer may race to land
writes either after the CPU thinks it's got a coherent view, or after its
GTT entries have been redirected to point at the scratch page.  Either
result is unpleasant.
Signed-off-by: NEric Anholt <eric@anholt.net>

According to the docs, the ringbuffer is not allowed to wrap in the middle
of an instruction.

G45 PRM, Vol 1b, p101:
  While the “free space” wrap may allow commands to be wrapped around the
  end of the Ring Buffer, the wrap should only occur between commands.
  Padding (with NOP) may be required to follow this restriction.

Do as commanded.

[Having seen bug reports where there is evidence of split commands, but
apparently the GPU has continued on merrily before a bizarre and untimely
death, this may or may not fix a few random hangs.]
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
CC: Eric Anholt <eric@anholt.net>
Signed-off-by: NEric Anholt <eric@anholt.net>

There are several sources of unnecessary power consumption on Intel
graphics systems. The first is the LVDS clock. TFTs don't suffer from
persistence issues like CRTs, and so we can reduce the LVDS refresh rate
when the screen is idle. It will be automatically upclocked when
userspace triggers graphical activity. Beyond that, we can enable memory
self refresh. This allows the memory to go into a lower power state when
the graphics are idle. Finally, we can drop some clocks on the gpu
itself. All of these things can be reenabled between frames when GPU
activity is triggered, and so there should be no user visible graphical
changes.
Signed-off-by: NJesse Barnes <jesse.barnes@intel.com>
Signed-off-by: NMatthew Garrett <mjg@redhat.com>
Signed-off-by: NEric Anholt <eric@anholt.net>

Remember to release the local reference if we fail to wait on
the rendering.

(Also whilst in the vicinity add some whitespace so that the phasing of
the operations is clearer.)
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

The lack of a proper LRU was partially worked around by taking the fence
from the object containing the oldest seqno.  But if there are multiple
objects inactive, then they don't have seqnos and the first fence reg
among them would be chosen.  If you were trying to copy data between two
mappings, this could result in each page fault stealing the fence from
the other argument, and your application hanging.

https://bugs.freedesktop.org/show_bug.cgi?id=23566
https://bugs.freedesktop.org/show_bug.cgi?id=23220
https://bugs.freedesktop.org/show_bug.cgi?id=23253
https://bugs.freedesktop.org/show_bug.cgi?id=23366

Cc: Stable Team <stable@kernel.org>
Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>

Several functions in the GEM kernel API used int as handle type, but
user API has it __u32 which is also the intended type.

Replace int with u32.
Signed-off-by: NPekka Paalanen <pq@iki.fi>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NEric Anholt <eric@anholt.net>

As of 52dc7d32, we could leave an old
linear GTT mapping in place, so that apps trying to GTT-mapped write in
tiled data wouldn't get the fence added, and garbage would get displayed.
Signed-off-by: NEric Anholt <eric@anholt.net>

As we call unmap_mapping_range() twice in identical fashion, refactor
and attempt to explain why we need to call unmap_mapping_range().
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

Unitialized fence register could leads to corrupted display. Problem
encountered on MacBooks (revision 1 and 2), directly booting from EFI
or through BIOS emulation.

(bug #21710 at freedestop.org)
Signed-off-by: NGrégoire Henry <henry@pps.jussieu.fr>
Signed-off-by: NEric Anholt <eric@anholt.net>

Signed-off-by: NKrzysztof Halasa <khc@pm.waw.pl>
Signed-off-by: NEric Anholt <eric@anholt.net>

It hasn't been used in ages, and having the user tell your how much
memory is being freed at free time is a recipe for disaster even if it
was ever used.
Signed-off-by: NEric Anholt <eric@anholt.net>

The fence register value also depends upon the stride of the object, so we
need to clear the fence if that is changed as well.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
[anholt: Added 8xx and 965 paths, and renamed the confusing
i915_gem_object_tiling_ok function to i915_gem_object_fence_offset_ok]
Signed-off-by: NEric Anholt <eric@anholt.net>

With the work by Jesse Barnes to eliminate allocation of fences during
execbuffer, it becomes possible to write to the scan-out buffer with it
never acquiring a fence (simply by only ever writing to the object using
tiled GPU commands and never writing to it via the GTT). So for pre-i965
chipsets which require fenced access for tiled scan-out buffers, we need
to obtain a fence register.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

After performing an operation over the page list for a buffer retrieved by
i915_gem_object_get_pages() the pages need to be returned with
i915_gem_object_put_pages(). This was not being observed for the phys
objects which were thus leaking references to their backing pages.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
CC: Dave Airlie <airlied@gmail.com>
Signed-off-by: NEric Anholt <eric@anholt.net>

To differentiate between encountering an out-of-memory error with running
out of space in the aperture, use ENOSPC for the later.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

The batch buffer may be shared with another read buffer, so we should not
ignore any previously set domains, but just or in the command domain (and
check that the buffer is not writable).
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

By sending a broken execbuffer (its length was not suitably aligned) I
triggered an operation upon a freed object. The invalid alignment was
discovered after updating the write_domain on the object but before the
object was placed on the active queue. So during the unwind process
following the error, the now freed object attempts to flush its
non-existent, but outstanding, GPU writes causing this use-after-free.

[drm:i915_dispatch_gem_execbuffer] *ERROR* alignment
[drm:i915_gem_execbuffer] *ERROR* dispatch failed -22
WARNING: at lib/kref.c:43 warn_slowpath_null+0x10/0x15()
Modules linked in:
Pid: 4552, comm: lt-csi-drm Not tainted 2.6.30-rc6 #423
Call Trace:
 [<c0119ef3>] warn_slowpath_fmt+0x57/0x6d
 [<c014de24>] ? get_pageblock_migratetype+0x18/0x1e
 [<c014e8fd>] ? free_hot_page+0xa/0xc
 [<c014e915>] ? __free_pages+0x16/0x1f
 [<c0153ebf>] ? shmem_truncate_range+0x63e/0x656
 [<c015fb2f>] ? slob_page_alloc+0x146/0x1c8
 [<c0119f19>] warn_slowpath_null+0x10/0x15
 [<c01f55f2>] kref_get+0x1b/0x21
 [<c02605db>] i915_gem_object_move_to_active+0x1f/0x56
 [<c0261302>] i915_add_request+0x156/0x19a
 [<c026136e>] i915_gem_object_flush_gpu_write_domain+0x28/0x3f
 [<c0261eca>] i915_gem_object_unbind+0x4a/0x124
 [<c0261fd7>] i915_gem_free_object+0x33/0x9b
 [<c0250d6b>] drm_gem_object_free+0x28/0x4a
 [<c0250d43>] ? drm_gem_object_free+0x0/0x4a
 [<c01f55ce>] kref_put+0x38/0x41
 [<c0250cbf>] drm_gem_object_unreference+0x11/0x13
 [<c0250d06>] drm_gem_object_handle_unreference+0x1e/0x21
 [<c0250d13>] drm_gem_object_release_handle+0xa/0xe
 [<c01f3e6b>] idr_for_each+0x5f/0x98
 [<c0250d09>] ? drm_gem_object_release_handle+0x0/0xe
 [<c0250daf>] drm_gem_release+0x22/0x34
 [<c025046f>] drm_release+0x1e8/0x3c4
 [<c0162d25>] __fput+0xaf/0x146
 [<c0162dce>] fput+0x12/0x14
 [<c01605ef>] filp_close+0x48/0x52
 [<c011b182>] put_files_struct+0x57/0x9b
 [<c011b1e4>] exit_files+0x1e/0x20
 [<c011c6b6>] do_exit+0x16d/0x511
 [<c03704ab>] ? __schedule+0x3d4/0x3e5
 [<c0103f0d>] ? handle_irq+0xd/0x69
 [<c011caa7>] do_group_exit+0x4d/0x73
 [<c011cae0>] sys_exit_group+0x13/0x17
 [<c010268c>] sysenter_do_call+0x12/0x2b
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

Update interrupt handling methods for IGDNG with new registers
for display and graphics interrupt functions. As we won't use
irq-based vblank sync in dri2, so display interrupt on new chip
will be used for hotplug only in future.
Signed-off-by: NZhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: NEric Anholt <eric@anholt.net>

keithp didn't like the original 20ms plan because a cooperative client could
be starved by an uncooperative client.  There may even have been problems
with cooperative clients versus cooperative clients.  So keithp changed
throttle to just wait for the second to last seqno emitted by that client.
It worked well, until we started getting more round-trips to the server
due to DRI2 -- the server throttles in BlockHandler, and so if you did more
than one round trip after finishing your frame, you'd end up unintentionally
syncing to the swap.

Fix this by keeping track of the client's requests, so the client can wait
when it has an outstanding request over 20ms old.  This should have
non-starving behavior, good behavior in the presence of restarts, and less
waiting.  Improves high-settings openarena performance on my GM45 by 50%.
Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>

This could be triggered by a gtt mapping fault on 965 that decides to
remove the fence from another object that happens to be active currently.
Since the other object doesn't rely on the fence reg for its execution, we
don't wait for it to finish.  We'll soon be not waiting on 915 most of the
time as well, so just drop the BUG_ON.
Signed-off-by: NEric Anholt <eric@anholt.net>