stb_truetype: fix incorrect antialiasing computation in v2 rasterizer, and handle certain cases where math blew up