Replace size check with more meaningful pubkey check

It does not make sense to check the size because this
function can be used in other contexts than in TLS-1.3 and
the value might not be padded to the size of p.

However it makes sense to do the partial pubkey check because
there is no valid reason having the pubkey value outside the
1 < pubkey < p-1 bounds.

Fixes #15465
Reviewed-by: NPaul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/17630)

crypto/dh/dh_key.c

@@ -375,20 +375,17 @@ int ossl_dh_buf2key(DH *dh, const unsigned char *buf, size_t len)
     int err_reason = DH_R_BN_ERROR;
     BIGNUM *pubkey = NULL;
     const BIGNUM *p;
-    size_t p_size;
+    int ret;
 
     if ((pubkey = BN_bin2bn(buf, len, NULL)) == NULL)
         goto err;
     DH_get0_pqg(dh, &p, NULL, NULL);
-    if (p == NULL || (p_size = BN_num_bytes(p)) == 0) {
+    if (p == NULL || BN_num_bytes(p) == 0) {
         err_reason = DH_R_NO_PARAMETERS_SET;
         goto err;
     }
-    /*
-     * As per Section 4.2.8.1 of RFC 8446 fail if DHE's
-     * public key is of size not equal to size of p
-     */
-    if (BN_is_zero(pubkey) || p_size != len) {
+    /* Prevent small subgroup attacks per RFC 8446 Section 4.2.8.1 */
+    if (!ossl_dh_check_pub_key_partial(dh, pubkey, &ret)) {
         err_reason = DH_R_INVALID_PUBKEY;
         goto err;
     }