Fix hostname validation in the command-line tool to honour negative return values.

Specifically, an ASN.1 NumericString in the certificate CN will fail UTF-8 conversion
and result in a negative return value, which the "x509 -checkhost" command-line option
incorrectly interpreted as success.

Also update X509_check_host docs to reflect reality.

Thanks to Sean Burford (Google) for reporting this issue.
Reviewed-by: NRichard Levitte <levitte@openssl.org>

apps/apps.c

@@ -2780,7 +2780,7 @@ void print_cert_checks(BIO *bio, X509 *x,
         return;
     if (checkhost) {
         BIO_printf(bio, "Hostname %s does%s match certificate\n",
-                   checkhost, X509_check_host(x, checkhost, 0, 0, NULL)
+                   checkhost, X509_check_host(x, checkhost, 0, 0, NULL) == 1
                    ? "" : " NOT");
     }
 

crypto/x509v3/v3_utl.c

@@ -889,8 +889,13 @@ static int do_check_string(ASN1_STRING *a, int cmp_type, equal_fn equal,
         int astrlen;
         unsigned char *astr;
         astrlen = ASN1_STRING_to_UTF8(&astr, a);
-        if (astrlen < 0)
+        if (astrlen < 0) {
+            /*
+             * -1 could be an internal malloc failure or a decoding error from
+             * malformed input; we can't distinguish.
+             */
             return -1;
+        }
         rv = equal(astr, astrlen, (unsigned char *)b, blen, flags);
         if (rv > 0 && peername)
             *peername = BUF_strndup((char *)astr, astrlen);

doc/crypto/X509_check_host.pod

@@ -109,9 +109,12 @@ but would not match a peer certificate with a DNS name of
 =head1 RETURN VALUES
 
 The functions return 1 for a successful match, 0 for a failed match
-and -1 for an internal error: typically a memory allocation failure.
+and -1 for an internal error: typically a memory allocation failure
+or an ASN.1 decoding error.
 
-X509_check_ip_asc() can also return -2 if the IP address string is malformed.
+All functions can also return -2 if the input is malformed. For example,
+X509_check_host() returns -2 if the provided B<name> contains embedded
+NULs.
 
 =head1 NOTES