Extends the fix in f2f9a2cb to also
workaround permission issues noticed by people using AGP systems.

Cc: stable@vger.kernel.org # 3.16: f2f9a2cb: drm/nouveau: fix regression
Cc: stable@vger.kernel.org # 3.16+
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>