drm/vmwgfx: Work around mode set failure in 2D VMs

In a low-memory 2D VM, fbdev can take up a large percentage of
available memory, making them unavailable for other DRM clients.

Since we do not take fbdev into account when filtering modes,
we end up claiming to support more modes than we actually do.

As a result, users get a black screen when setting a mode too
large for current available memory.  In a low-memory VM
configuration, users can get a black screen for a mode as low
as 1024x768.

The current mode filtering mechanism keys off of
SVGA_REG_SUGGESTED_GBOBJECT_MEM_SIZE_KB, i.e. the maximum amount
of surface memory we have.  Since this value is a performance
suggestion, not a hard limit, and since there should not be much
of a performance impact for a 2D VM, rather than filtering out
more modes, we will just allow ourselves to exceed the SVGA's
performance suggestion.

Also changed assumed bpp to 32 from 16 to make sure we can
actually support all the modes listed.
Signed-off-by: NSinclair Yeh <syeh@vmware.com>
Reviewed-by: NThomas Hellstrom <thellstrom@vmware.com>
Cc: <stable@vger.kernel.org>

drivers/gpu/drm/vmwgfx/vmwgfx_drv.c

@@ -706,6 +706,13 @@ static int vmw_driver_load(struct drm_device *dev, unsigned long chipset)
 			vmw_read(dev_priv,
 				 SVGA_REG_SUGGESTED_GBOBJECT_MEM_SIZE_KB);
 
+		/*
+		 * Workaround for low memory 2D VMs to compensate for the
+		 * allocation taken by fbdev
+		 */
+		if (!(dev_priv->capabilities & SVGA_CAP_3D))
+			mem_size *= 2;
+
 		dev_priv->max_mob_pages = mem_size * 1024 / PAGE_SIZE;
 		dev_priv->prim_bb_mem =
 			vmw_read(dev_priv,

drivers/gpu/drm/vmwgfx/vmwgfx_kms.c

@@ -1553,14 +1553,7 @@ int vmw_du_connector_fill_modes(struct drm_connector *connector,
 		DRM_MODE_FLAG_NHSYNC | DRM_MODE_FLAG_PVSYNC)
 	};
 	int i;
-	u32 assumed_bpp = 2;
-
-	/*
-	 * If using screen objects, then assume 32-bpp because that's what the
-	 * SVGA device is assuming
-	 */
-	if (dev_priv->active_display_unit == vmw_du_screen_object)
-		assumed_bpp = 4;
+	u32 assumed_bpp = 4;
 
 	if (dev_priv->active_display_unit == vmw_du_screen_target) {
 		max_width  = min(max_width,  dev_priv->stdu_max_width);