Greatly simplifies a number of things, particularly once per-client GPU
address spaces are involved.

May add this back later once I know what things'll look like.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>