Add a sanity check on the length of pkeyutl inputs

When signing or verifying a file using pkeyutl the input is supposed to
be a hash. Some algorithms sanity check the length of the input, while
others don't and silently truncate. To avoid accidents we check that the
length of the input looks sane.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6284)

CHANGES

@@ -8,6 +8,10 @@
  release branch.
 
  Changes between 1.1.0h and 1.1.1 [xx XXX xxxx]
+  *) Enforce checking in the pkeyutl command line app to ensure that the input
+     length does not exceed the maximum supported digest length when performing
+     a sign, verify or verifyrecover operation.
+     [Matt Caswell]
 
   *) SSL_MODE_AUTO_RETRY is enabled by default. Applications that use blocking
      I/O in combination with something like select() or poll() will hang. This

apps/pkeyutl.c

@@ -282,7 +282,7 @@ int pkeyutl_main(int argc, char **argv)
         buf_inlen = bio_to_mem(&buf_in, keysize * 10, in);
         if (buf_inlen < 0) {
             BIO_printf(bio_err, "Error reading input Data\n");
-            exit(1);
+            goto end;
         }
         if (rev) {
             size_t i;
@@ -296,6 +296,16 @@ int pkeyutl_main(int argc, char **argv)
         }
     }
 
+    /* Sanity check the input */
+    if (buf_inlen > EVP_MAX_MD_SIZE
+            && (pkey_op == EVP_PKEY_OP_SIGN
+                || pkey_op == EVP_PKEY_OP_VERIFY
+                || pkey_op == EVP_PKEY_OP_VERIFYRECOVER)) {
+        BIO_printf(bio_err,
+                   "Error: The input data looks too long to be a hash\n");
+        goto end;
+    }
+
     if (pkey_op == EVP_PKEY_OP_VERIFY) {
         rv = EVP_PKEY_verify(ctx, sig, (size_t)siglen,
                              buf_in, (size_t)buf_inlen);

doc/man1/pkeyutl.pod

@@ -38,8 +38,8 @@ B<openssl> B<pkeyutl>
 
 =head1 DESCRIPTION
 
-The B<pkeyutl> command can be used to perform public key operations using
-any supported algorithm.
+The B<pkeyutl> command can be used to perform low level public key operations
+using any supported algorithm.
 
 =head1 OPTIONS
 
@@ -99,17 +99,17 @@ Reverse the order of the input buffer. This is useful for some libraries
 
 =item B<-sign>
 
-Sign the input data and output the signed result. This requires
-a private key.
+Sign the input data (which must be a hash) and output the signed result. This
+requires a private key.
 
 =item B<-verify>
 
-Verify the input data against the signature file and indicate if the
-verification succeeded or failed.
+Verify the input data (which must be a hash) against the signature file and
+indicate if the verification succeeded or failed.
 
 =item B<-verifyrecover>
 
-Verify the input data and output the recovered data.
+Verify the input data (which must be a hash) and output the recovered data.
 
 =item B<-encrypt>
 
@@ -184,20 +184,25 @@ and its implementation. The OpenSSL operations and options are indicated below.
 Unless otherwise mentioned all algorithms support the B<digest:alg> option
 which specifies the digest in use for sign, verify and verifyrecover operations.
 The value B<alg> should represent a digest name as used in the
-EVP_get_digestbyname() function for example B<sha1>.
-This value is used only for sanity-checking the lengths of data passed in to
-the B<pkeyutl> and for creating the structures that make up the signature
-(e.g. B<DigestInfo> in RSASSA PKCS#1 v1.5 signatures).
-In case of RSA, ECDSA and DSA signatures, this utility
-will not perform hashing on input data but rather use the data directly as
-input of signature algorithm. Depending on key type, signature type and mode
-of padding, the maximum acceptable lengths of input data differ. In general,
-with RSA the signed data can't be longer than the key modulus, in case of ECDSA
-and DSA the data shouldn't be longer than field size, otherwise it will be
-silently truncated to field size.
-
-In other words, if the value of digest is B<sha1> the input should be 20 bytes
-long binary encoding of SHA-1 hash function output.
+EVP_get_digestbyname() function for example B<sha1>. This value is not used to
+hash the input data. It is used (by some algorithms) for sanity-checking the
+lengths of data passed in to the B<pkeyutl> and for creating the structures that
+make up the signature (e.g. B<DigestInfo> in RSASSA PKCS#1 v1.5 signatures).
+
+This utility does not hash the input data but rather it will use the data
+directly as input to the signature algorithm. Depending on the key type,
+signature type, and mode of padding, the maximum acceptable lengths of input
+data differ. The signed data can't be longer than the key modulus with RSA. In
+case of ECDSA and DSA the data shouldn't be longer than the field
+size, otherwise it will be silently truncated to the field size. In any event
+the input size must not be larger than the largest supported digest size.
+
+In other words, if the value of digest is B<sha1> the input should be the 20
+bytes long binary encoding of the SHA-1 hash function output.
+
+The Ed25519 and Ed448 signature algorithms are not supported by this utility.
+They accept non-hashed input, but this utility can only be used to sign hashed
+input.
 
 =head1 RSA ALGORITHM