Linux returns 0 if no conversion was made, while OS X and presumably
the BSDs return EINVAL.  The OS X convention rejects more invalid
inputs, so convert to it and adjust the test case.

Windows returns 1 from strtoul and strtoull (instead of -1) for
negative out-of-range input; fix it up.
Reported-by: NPeter Maydell <peter.maydell@linaro.org>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>