For awhile now, many of the GEM code paths have allocated page or
object arrays with the slab allocator.  This is nice and fast, but
won't work well if memory is fragmented, since the slab allocator works
with physically contiguous memory (i.e. order > 2 allocations are
likely to fail fairly early after booting and doing some work).

This patch works around the issue by falling back to vmalloc for
>PAGE_SIZE allocations.  This is ugly, but much less work than chaining
a bunch of pages together by hand (suprisingly there's not a bunch of
generic kernel helpers for this yet afaik).  vmalloc space is somewhat
precious on 32 bit kernels, but our allocations shouldn't be big enough
to cause problems, though they're routinely more than a page.

Note that this patch doesn't address the unchecked
alloc-based-on-ioctl-args in GEM; that needs to be fixed in a separate
patch.

Also, I've deliberately ignored the DRM's "area" junk.  I don't think
anyone actually uses it anymore and I'm hoping it gets ripped out soon.

[Updated: removed size arg to new free function.  We could unify the
free functions as well once the DRM mem tracking is ripped out.]

fd.o bug #20152 (part 1/3)
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NEric Anholt <eric@anholt.net>

We might sleep here anyway so I hope an extra uncached read is ok to
add.

In #20896 we found that vbetool clobbers the IER.  In KMS mode this is
particularly bad since we don't set the interrupt regs late (in
EnterVT), so we'd fail to get *any* interrupts at all after X started
(since some distros have scripts that call vbetool at X startup
apparently).

So this patch checks IER at wait_request time, and re-enables
interrupts if it's been clobbered.  In a proper config this check
should never be triggered.

This is really a distro issue, but having a sanity check is nice, as
long as it doesn't have a real performance hit.
Tested-by: NMateusz Kaduk <mateusz.kaduk@gmail.com>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>
[anholt: Moved the check inside of the sleeping case to avoid perf cost]
Signed-off-by: NEric Anholt <eric@anholt.net>

Signed-off-by: NWu Fengguang <fengguang.wu@intel.com>
Signed-off-by: NEric Anholt <eric@anholt.net>

regression caused by commit 5e118f41:
i915_gem_object_move_to_inactive() should be called in task context,
as it calls fput();

Signed-off-by: Shaohua Li<shaohua.li@intel.com>
[anholt: Add more detail to the comment about the lock break that's added]
Signed-off-by: NEric Anholt <eric@anholt.net>

Save the bit 17 state of the pages when freeing the page list, and
reswizzle them if necessary when rebinding the pages (in case they were
swapped out).  Since we have userland with expectations that the swizzle
enums let it pread and pwrite contents accurately, we can't expose a new
swizzle enum for bit 17 (which it would have to GTT map to handle), so we
handle it down in pread and pwrite by swizzling the copy when bit 17 of the
page address is set.
Signed-off-by: NEric Anholt <eric@anholt.net>

Otherwise, the results of our read didn't show up when we were faulting in
the page being read into (as happened with a testcase reading into a big
stack area).  Likely accounts for some conformance test failures.
Signed-off-by: NEric Anholt <eric@anholt.net>

i915_gem_put_relocs_to_user returned an uninitialized value which
got returned to userspace. This caused libdrm in my setup to never
get out of a do{}while() loop retrying i915_gem_execbuffer.

result was hanging X, overheating of cpu and 2-3gb of logfile-spam.

This patch adresses the issue by
 1. initializing vars in this file where necessary
 2. correcting wrongly interpreted return values of copy_[from/to]_user
Signed-off-by: NFlorian Mickler <florian@mickler.org>
[anholt: cleanups of unnecessary changes, consistency in APIs]
Signed-off-by: NEric Anholt <eric@anholt.net>

We create a debugfs node (i915_ringbuffer_data) to expose a hex dump
of the ring buffer itself.  We also expose another debugfs node
(i915_ringbuffer_info) with information on the state (i.e. head, tail
addresses) of the ringbuffer.

For batchbuffer dumping, we look at the device's active_list, dumping
each object which has I915_GEM_DOMAIN_COMMAND in its read
domains. This is all exposed through the dri/i915_batchbuffers debugfs
file with a header for each object (giving the objects gtt_offset so
that it can be matched against the offset given in the
BATCH_BUFFER_START command.
Signed-off-by: NBen Gamari <bgamari@gmail.com>
Signed-off-by: NCarl Worth <cworth@cworth.org>
Signed-off-by: NEric Anholt <eric@anholt.net>

This is a baby-step in the direction of having finer-grained
locking than the struct_mutex. Specifically, this will enable
new debugging code to read the active list for printing out
GPU state when the GPU is wedged, (while the struct_mutex is
held, of course).
Signed-off-by: NCarl Worth <cworth@cworth.org>
[anholt: indentation fix]
Signed-off-by: NEric Anholt <eric@anholt.net>

Indicates something is wrong with the mapping; and apparently triggers
in current kernels.
Signed-off-by: NJesse Barnes <jbarnes@virtuosugeek.org>
Signed-off-by: NEric Anholt <eric@anholt.net>

This fixes all the tiling problems with the 2d ddx. glxgears still doesn't work.
Changes:

- fix a copy&paste error in i8xx fence reg setup. It resulted in an at most a
  512KB offset of the fence reg window, so was only visible sometimes.
- add tests for stride and object size constrains (also for i915 and 1965 class
  hw). Userspace seems to have an of-by-one bug there, which changes the fence
  size by at most 512KB due to an overflow.
- because i8xx hw is quite old (and therefore not as well-tested) I left 2 debug
  WARN_ONs in the i8xx fence reg setup code to hopefully catch any further
  overflows in the bit-fields. Lastly there's one small change to make the
  alignment checks more consistent.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=20289Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: NEric Anholt <eric@anholt.net>

This produced a warning on my build, not sure why super-warning-man didn't
notice this one, its much worse than the %z one.
Signed-off-by: NDave Airlie <airlied@redhat.com>

agp_chipset_flush() is for flushing the intel GMCH write cache via the
IFP, these two uses are for when we're getting the object into the cpu
READ domain, and thus should not be needed. This confused me when I was
getting my head around the code.

With thanks to airlied for helping me check my mental picture of how the
flushes and clflushes are supposed to be used.
Signed-off-by: NOwain G. Ainsworth <oga@openbsd.org>
Signed-off-by: NEric Anholt <eric@anholt.net>

Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NKeith Packard <keithp@keithp.com>

This introduces allocation in the batch submission path that wasn't there
previously, but these are compatibility paths so we care about simplicity
more than performance.

kernel.org bug #12419.
Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NKeith Packard <keithp@keithp.com>
Acked-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Like the GTT pwrite path fix, this uses an optimistic path and a
fallback to get_user_pages.  Note that this means we have to stop using
vfs_write and roll it ourselves.
Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>

We've wanted this for a few consumers that touch the pages directly (such as
the following commit), which have been doing the refcounting outside of
get/put pages.
Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>

Since the pagefault path determines that the lock order we use has to be
mmap_sem -> struct_mutex, we can't allow page faults to occur while the
struct_mutex is held.  To fix this in pwrite, we first try optimistically to
see if we can copy from user without faulting.  If it fails, fall back to
using get_user_pages to pin the user's memory, and map those pages
atomically when copying it to the GPU.
Signed-off-by: NEric Anholt <eric@anholt.net>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>

agp_chipset_flush() is for flushing the intel GMCH write cache via the
IFP, these two uses are for when we're getting the object into the cpu
READ domain, and thus should not be needed. This confused me when I was
getting my head around the code.

With thanks to airlied for helping me check my mental picture of how the
flushes and clflushes are supposed to be used.
Signed-off-by: NOwain G. Ainsworth <oga@openbsd.org>
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Once upon a time, the DRM made the distinction between the drm_map
data structure exchanged with user space and the drm_local_map used
in the kernel.

For some reasons, while the BSD port still has that "feature", the
linux part abused drm_map for kernel internal usage as the local
map only existed as a typedef of the struct drm_map.

This patch fixes it by declaring struct drm_local_map separately
(though its content is currently identical to the userspace variant),
and changing the kernel code to only use that, except when it's a
user<->kernel interface (ie. ioctl).

This allows subsequent changes to the in-kernel format

I've also replaced the use of drm_local_map_t with struct drm_local_map
in a couple of places. Mostly by accident but they are the same (the
former is a typedef of the later) and I have some remote plans and
half finished patch to completely kill the drm_local_map_t typedef
so I left those bits in.
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDave Airlie <airlied@linux.ie>

The last 8 fence registers sit at a different offset, so when we went to set
fence number 8 in the lower offset, we instead set PGETBL_CTL, and the GPU
got all sorts of angry at us.

fd.o bug #20567.  Easily reproducible by running glxgears and killing it about
6 times.
Signed-off-by: NEric Anholt <eric@anholt.net>

The i915 also uses the fence registers for GPU access to tiled buffers so
we cannot reallocate one whilst it is on the active list. By performing a
LRU scan of the fenced buffers we also avoid waiting the possibility of
waiting on a pinned, or otherwise unusable, buffer.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

We need to check and report if there are no available fences - or else we
spin endlessly waiting for a buffer to magically unpin itself.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

As we may steal the fence register of an unpinned buffer for another,
every time we repin the buffer we need to recheck whether it needs to be
allocated a fence.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

If we wait upon a request and successfully unbind a buffer occupying a
fence register, then that slot will be freed and cause a NULL derefrence
upon rescanning.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>

Prevents formatting nasty as below:

[drm:i915_gem_object_pin] *ERROR* Failure to bind: -12<3>[drm:i915_gem_evict_something] *ERROR* inactive empty 1 request empty 1 flushing empty 1
Signed-off-by: NKyle McMartin <kyle@redhat.com>
Signed-off-by: NEric Anholt <eric@anholt.net>

If userspace passes an object list with the same object appearing more
than once, we end up hitting the BUG_ON() in
i915_gem_object_set_to_gpu_domain() as it gets called a second time
for the same object.
Signed-off-by: NKristian Høgsberg <krh@redhat.com>
Signed-off-by: NEric Anholt <eric@anholt.net>

Impact: standardize IO on cached ops

On modern CPUs it is almost always a bad idea to use non-temporal stores,
as the regression in this commit has shown it:

  30d697fa: x86: fix performance regression in write() syscall

The kernel simply has no good information about whether using non-temporal
stores is a good idea or not - and trying to add heuristics only increases
complexity and inserts fragility.

The regression on cached write()s took very long to be found - over two
years. So dont take any chances and let the hardware decide how it makes
use of its caches.

The only exception is drivers/gpu/drm/i915/i915_gem.c: there were we are
absolutely sure that another entity (the GPU) will pick up the dirty
data immediately and that the CPU will not touch that data before the
GPU will.

Also, keep the _nocache() primitives to make it easier for people to
experiment with these details. There may be more clear-cut cases where
non-cached copies can be used, outside of filemap.c.

Cc: Salman Qazi <sqazi@google.com>
Cc: Nick Piggin <npiggin@suse.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NIngo Molnar <mingo@elte.hu>

Impact: cleanup, enable future change

Add a 'total bytes copied' parameter to __copy_from_user_*nocache(),
and update all the callsites.

The parameter is not used yet - architecture code can use it to
more intelligently decide whether the copy should be cached or
non-temporal.

Cc: Salman Qazi <sqazi@google.com>
Cc: Nick Piggin <npiggin@suse.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NIngo Molnar <mingo@elte.hu>

This snuck in when I wrote phys object support.
Signed-off-by: NDave Airlie <airlied@redhat.com>

dev_priv->hw_status_page can be NULL, if i915_gem_retire_requests()
is called from i915_gem_busy_ioctl().

Signed-off-by Karsten Wiese <fzu@wemgehoertderstaat.de>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The object is dereferenced before the NULL check. Oops.

Fixes http://bugs.freedesktop.org/show_bug.cgi?id=20235Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This ensures that the user gets the latest information from the hardware
on whether the buffer is busy, potentially reducing the working set of objects
that the user chooses.
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDave Airlie <airlied@redhat.com>

In the KMS case, we need to suspend/resume GEM as well.  So on suspend, make
sure we idle GEM and stop any new rendering from coming in, and on resume,
re-init the framebuffer and clear the suspended flag.
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDave Airlie <airlied@redhat.com>

The problem was that object_set_to_gpu_domain would set the new write_domains
that are getting set by this batchbuffer, then the accumulated flushes required
for all the objects in preparation for this batchbuffer were posted, and the
brand new write domain would get cleared by the flush being posted.  Instead,
hang on to the new (or old if we're not changing it) value and set it after
the flush is queued.

Results from this noticably included conformance test failures from reads
shortly after writes (where the new write domain had been lost and thus not
flushed and waited on), but is a suspected cause of hangs in some apps when
a write domain is lost on a buffer that gets reused for instruction or
commmand state.
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDave Airlie <airlied@redhat.com>

While not strictly required, it helped while thinking about the following
change.  This change should be invariant.
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This fixes potential fault at fault time if the object was unreferenced
while the mapping still existed.  Now, while the mmap_offset only lives
for the lifetime of the object, the object also stays alive while a vma
exists that needs it.
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDave Airlie <airlied@redhat.com>

If we fail to create the ringbuffer, then we need to cleanup the allocated
hws.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDave Airlie <airlied@linux.ie>

A missing unpin on the error path.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NEric Anholt <eric@anholt.net>
Signed-off-by: NDave Airlie <airlied@linux.ie>