Skip to content

cloud-kernel
cloud-kernel

openanolis / cloud-kernel
大约 1 年 前同步成功

通知 157
Star 36
Fork 7
cloud-kernel
cloud-kernel

体验新版 GitCode，发现更多精彩内容 >>

提交 bbce2ad2 编写于 作者: L Linus Torvalds
浏览文件
操作

Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6 

Pull crypto updates from Herbert Xu:
 "Here is the crypto update for 4.8:

  API:
   - first part of skcipher low-level conversions
   - add KPP (Key-agreement Protocol Primitives) interface.

  Algorithms:
   - fix IPsec/cryptd reordering issues that affects aesni
   - RSA no longer does explicit leading zero removal
   - add SHA3
   - add DH
   - add ECDH
   - improve DRBG performance by not doing CTR by hand

  Drivers:
   - add x86 AVX2 multibuffer SHA256/512
   - add POWER8 optimised crc32c
   - add xts support to vmx
   - add DH support to qat
   - add RSA support to caam
   - add Layerscape support to caam
   - add SEC1 AEAD support to talitos
   - improve performance by chaining requests in marvell/cesa
   - add support for Araneus Alea I USB RNG
   - add support for Broadcom BCM5301 RNG
   - add support for Amlogic Meson RNG
   - add support Broadcom NSP SoC RNG"

* 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (180 commits)
  crypto: vmx - Fix aes_p8_xts_decrypt build failure
  crypto: vmx - Ignore generated files
  crypto: vmx - Adding support for XTS
  crypto: vmx - Adding asm subroutines for XTS
  crypto: skcipher - add comment for skcipher_alg->base
  crypto: testmgr - Print akcipher algorithm name
  crypto: marvell - Fix wrong flag used for GFP in mv_cesa_dma_add_iv_op
  crypto: nx - off by one bug in nx_of_update_msc()
  crypto: rsa-pkcs1pad - fix rsa-pkcs1pad request struct
  crypto: scatterwalk - Inline start/map/done
  crypto: scatterwalk - Remove unnecessary BUG in scatterwalk_start
  crypto: scatterwalk - Remove unnecessary advance in scatterwalk_pagedone
  crypto: scatterwalk - Fix test in scatterwalk_done
  crypto: api - Optimise away crypto_yield when hard preemption is on
  crypto: scatterwalk - add no-copy support to copychunks
  crypto: scatterwalk - Remove scatterwalk_bytes_sglen
  crypto: omap - Stop using crypto scatterwalk_bytes_sglen
  crypto: skcipher - Remove top-level givcipher interface
  crypto: user - Remove crypto_lookup_skcipher call
  crypto: cts - Convert to skcipher
  ...
上级 0f776dc3 0f95e2ff
展开全部 隐藏空白更改
内联 并排
Showing with 19350 addition and 4261 deletion
Documentation/DocBook/crypto-API.tmpl
浏览文件 @ bbce2ad2
......@@ -440,8 +440,8 @@
The type flag specifies the type of the cipher algorithm.
The caller usually provides a 0 when the caller wants the
default handling. Otherwise, the caller may provide the
following selections which match the the aforementioned
cipher types:
following selections which match the aforementioned cipher
types:
</para>
<itemizedlist>
......
Documentation/crypto/asymmetric-keys.txt
浏览文件 @ bbce2ad2
......@@ -76,7 +76,7 @@ the criterion string:
Looking in /proc/keys, the last 8 hex digits of the key fingerprint are
displayed, along with the subtype:
1a39e171 I----- 1 perm 3f010000 0 0 asymmetri modsign.0: DSA 5acc2142 []
1a39e171 I----- 1 perm 3f010000 0 0 asymmetric modsign.0: DSA 5acc2142 []
=========================
......
Documentation/devicetree/bindings/rng/brcm,bcm2835.txt
浏览文件 @ bbce2ad2
......@@ -2,7 +2,8 @@ BCM2835 Random number generator
Required properties:
- compatible : should be "brcm,bcm2835-rng"
- compatible : should be "brcm,bcm2835-rng" or "brcm,bcm-nsp-rng" or
"brcm,bcm5301x-rng"
- reg : Specifies base physical address and size of the registers.
Example:
......@@ -11,3 +12,8 @@ rng {
compatible = "brcm,bcm2835-rng";
reg = <0x7e104000 0x10>;
};
rng@18033000 {
compatible = "brcm,bcm-nsp-rng";
reg = <0x18033000 0x14>;
};
MAINTAINERS
浏览文件 @ bbce2ad2
......@@ -3286,6 +3286,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git
T: git git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6.git
S: Maintained
F: Documentation/crypto/
F: Documentation/devicetree/bindings/crypto/
F: Documentation/DocBook/crypto-API.tmpl
F: arch/*/crypto/
F: crypto/
......@@ -5273,6 +5274,7 @@ M: Matt Mackall <mpm@selenic.com>
M: Herbert Xu <herbert@gondor.apana.org.au>
L: linux-crypto@vger.kernel.org
S: Odd fixes
F: Documentation/devicetree/bindings/rng/
F: Documentation/hw_random.txt
F: drivers/char/hw_random/
F: include/linux/hw_random.h
......@@ -9318,7 +9320,8 @@ L: rtc-linux@googlegroups.com
S: Maintained
QAT DRIVER
M: Tadeusz Struk <tadeusz.struk@intel.com>
M: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
M: Salvatore Benedetto <salvatore.benedetto@intel.com>
L: qat-linux@intel.com
S: Supported
F: drivers/crypto/qat/
......
arch/arm/boot/dts/bcm-nsp.dtsi
浏览文件 @ bbce2ad2
......@@ -206,6 +206,11 @@
brcm,nand-has-wp;
};
rng: rng@33000 {
compatible = "brcm,bcm-nsp-rng";
reg = <0x33000 0x14>;
};
ccbtimer0: timer@34000 {
compatible = "arm,sp804";
reg = <0x34000 0x1000>;
......
arch/arm/crypto/ghash-ce-glue.c
浏览文件 @ bbce2ad2
......@@ -154,30 +154,23 @@ static int ghash_async_init(struct ahash_request *req)
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
struct crypto_shash *child = cryptd_ahash_child(cryptd_tfm);
if (!may_use_simd()) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_init(cryptd_req);
} else {
struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
struct crypto_shash *child = cryptd_ahash_child(cryptd_tfm);
desc->tfm = child;
desc->flags = req->base.flags;
return crypto_shash_init(desc);
}
desc->tfm = child;
desc->flags = req->base.flags;
return crypto_shash_init(desc);
}
static int ghash_async_update(struct ahash_request *req)
{
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
if (!may_use_simd()) {
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
if (!may_use_simd() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_update(cryptd_req);
......@@ -190,12 +183,12 @@ static int ghash_async_update(struct ahash_request *req)
static int ghash_async_final(struct ahash_request *req)
{
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
if (!may_use_simd()) {
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
if (!may_use_simd() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_final(cryptd_req);
......@@ -212,7 +205,8 @@ static int ghash_async_digest(struct ahash_request *req)
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
if (!may_use_simd()) {
if (!may_use_simd() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_digest(cryptd_req);
......
arch/arm64/boot/dts/freescale/fsl-ls1043a-rdb.dts
浏览文件 @ bbce2ad2
......@@ -49,6 +49,10 @@
/ {
model = "LS1043A RDB Board";
aliases {
crypto = &crypto;
};
};
&i2c0 {
......
arch/arm64/boot/dts/freescale/fsl-ls1043a.dtsi
浏览文件 @ bbce2ad2
......@@ -159,6 +159,49 @@
big-endian;
};
crypto: crypto@1700000 {
compatible = "fsl,sec-v5.4", "fsl,sec-v5.0",
"fsl,sec-v4.0";
fsl,sec-era = <3>;
#address-cells = <1>;
#size-cells = <1>;
ranges = <0x0 0x00 0x1700000 0x100000>;
reg = <0x00 0x1700000 0x0 0x100000>;
interrupts = <0 75 0x4>;
sec_jr0: jr@10000 {
compatible = "fsl,sec-v5.4-job-ring",
"fsl,sec-v5.0-job-ring",
"fsl,sec-v4.0-job-ring";
reg = <0x10000 0x10000>;
interrupts = <0 71 0x4>;
};
sec_jr1: jr@20000 {
compatible = "fsl,sec-v5.4-job-ring",
"fsl,sec-v5.0-job-ring",
"fsl,sec-v4.0-job-ring";
reg = <0x20000 0x10000>;
interrupts = <0 72 0x4>;
};
sec_jr2: jr@30000 {
compatible = "fsl,sec-v5.4-job-ring",
"fsl,sec-v5.0-job-ring",
"fsl,sec-v4.0-job-ring";
reg = <0x30000 0x10000>;
interrupts = <0 73 0x4>;
};
sec_jr3: jr@40000 {
compatible = "fsl,sec-v5.4-job-ring",
"fsl,sec-v5.0-job-ring",
"fsl,sec-v4.0-job-ring";
reg = <0x40000 0x10000>;
interrupts = <0 74 0x4>;
};
};
dcfg: dcfg@1ee0000 {
compatible = "fsl,ls1043a-dcfg", "syscon";
reg = <0x0 0x1ee0000 0x0 0x10000>;
......
arch/arm64/include/asm/io.h
浏览文件 @ bbce2ad2
......@@ -174,13 +174,15 @@ extern void __iomem *ioremap_cache(phys_addr_t phys_addr, size_t size);
#define iounmap __iounmap
/*
* io{read,write}{16,32}be() macros
* io{read,write}{16,32,64}be() macros
*/
#define ioread16be(p) ({ __u16 __v = be16_to_cpu((__force __be16)__raw_readw(p)); __iormb(); __v; })
#define ioread32be(p) ({ __u32 __v = be32_to_cpu((__force __be32)__raw_readl(p)); __iormb(); __v; })
#define ioread64be(p) ({ __u64 __v = be64_to_cpu((__force __be64)__raw_readq(p)); __iormb(); __v; })
#define iowrite16be(v,p) ({ __iowmb(); __raw_writew((__force __u16)cpu_to_be16(v), p); })
#define iowrite32be(v,p) ({ __iowmb(); __raw_writel((__force __u32)cpu_to_be32(v), p); })
#define iowrite64be(v,p) ({ __iowmb(); __raw_writeq((__force __u64)cpu_to_be64(v), p); })
/*
* Convert a physical pointer to a virtual kernel pointer for /dev/mem
......
arch/powerpc/crypto/Makefile
浏览文件 @ bbce2ad2
......@@ -9,9 +9,11 @@ obj-$(CONFIG_CRYPTO_MD5_PPC) += md5-ppc.o
obj-$(CONFIG_CRYPTO_SHA1_PPC) += sha1-powerpc.o
obj-$(CONFIG_CRYPTO_SHA1_PPC_SPE) += sha1-ppc-spe.o
obj-$(CONFIG_CRYPTO_SHA256_PPC_SPE) += sha256-ppc-spe.o
obj-$(CONFIG_CRYPT_CRC32C_VPMSUM) += crc32c-vpmsum.o
aes-ppc-spe-y := aes-spe-core.o aes-spe-keys.o aes-tab-4k.o aes-spe-modes.o aes-spe-glue.o
md5-ppc-y := md5-asm.o md5-glue.o
sha1-powerpc-y := sha1-powerpc-asm.o sha1.o
sha1-ppc-spe-y := sha1-spe-asm.o sha1-spe-glue.o
sha256-ppc-spe-y := sha256-spe-asm.o sha256-spe-glue.o
crc32c-vpmsum-y := crc32c-vpmsum_asm.o crc32c-vpmsum_glue.o
arch/powerpc/crypto/aes-spe-regs.h
浏览文件 @ bbce2ad2
......@@ -18,7 +18,7 @@
#define rLN r7 /* length of data to be processed */
#define rIP r8 /* potiner to IV (CBC/CTR/XTS modes) */
#define rKT r9 /* pointer to tweak key (XTS mode) */
#define rT0 r11 /* pointers to en-/decrpytion tables */
#define rT0 r11 /* pointers to en-/decryption tables */
#define rT1 r10
#define rD0 r9 /* data */
#define rD1 r14
......
arch/powerpc/crypto/crc32c-vpmsum_asm.S 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/powerpc/crypto/crc32c-vpmsum_glue.c 0 → 100644
浏览文件 @ bbce2ad2
#include <linux/crc32.h>
#include <crypto/internal/hash.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/string.h>
#include <linux/kernel.h>
#include <asm/switch_to.h>
#define CHKSUM_BLOCK_SIZE 1
#define CHKSUM_DIGEST_SIZE 4
#define VMX_ALIGN 16
#define VMX_ALIGN_MASK (VMX_ALIGN-1)
#define VECTOR_BREAKPOINT 512
u32 __crc32c_vpmsum(u32 crc, unsigned char const *p, size_t len);
static u32 crc32c_vpmsum(u32 crc, unsigned char const *p, size_t len)
{
unsigned int prealign;
unsigned int tail;
if (len < (VECTOR_BREAKPOINT + VMX_ALIGN) || in_interrupt())
return __crc32c_le(crc, p, len);
if ((unsigned long)p & VMX_ALIGN_MASK) {
prealign = VMX_ALIGN - ((unsigned long)p & VMX_ALIGN_MASK);
crc = __crc32c_le(crc, p, prealign);
len -= prealign;
p += prealign;
}
if (len & ~VMX_ALIGN_MASK) {
pagefault_disable();
enable_kernel_altivec();
crc = __crc32c_vpmsum(crc, p, len & ~VMX_ALIGN_MASK);
pagefault_enable();
}
tail = len & VMX_ALIGN_MASK;
if (tail) {
p += len & ~VMX_ALIGN_MASK;
crc = __crc32c_le(crc, p, tail);
}
return crc;
}
static int crc32c_vpmsum_cra_init(struct crypto_tfm *tfm)
{
u32 *key = crypto_tfm_ctx(tfm);
*key = 0;
return 0;
}
/*
* Setting the seed allows arbitrary accumulators and flexible XOR policy
* If your algorithm starts with ~0, then XOR with ~0 before you set
* the seed.
*/
static int crc32c_vpmsum_setkey(struct crypto_shash *hash, const u8 *key,
unsigned int keylen)
{
u32 *mctx = crypto_shash_ctx(hash);
if (keylen != sizeof(u32)) {
crypto_shash_set_flags(hash, CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
*mctx = le32_to_cpup((__le32 *)key);
return 0;
}
static int crc32c_vpmsum_init(struct shash_desc *desc)
{
u32 *mctx = crypto_shash_ctx(desc->tfm);
u32 *crcp = shash_desc_ctx(desc);
*crcp = *mctx;
return 0;
}
static int crc32c_vpmsum_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
u32 *crcp = shash_desc_ctx(desc);
*crcp = crc32c_vpmsum(*crcp, data, len);
return 0;
}
static int __crc32c_vpmsum_finup(u32 *crcp, const u8 *data, unsigned int len,
u8 *out)
{
*(__le32 *)out = ~cpu_to_le32(crc32c_vpmsum(*crcp, data, len));
return 0;
}
static int crc32c_vpmsum_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
return __crc32c_vpmsum_finup(shash_desc_ctx(desc), data, len, out);
}
static int crc32c_vpmsum_final(struct shash_desc *desc, u8 *out)
{
u32 *crcp = shash_desc_ctx(desc);
*(__le32 *)out = ~cpu_to_le32p(crcp);
return 0;
}
static int crc32c_vpmsum_digest(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
return __crc32c_vpmsum_finup(crypto_shash_ctx(desc->tfm), data, len,
out);
}
static struct shash_alg alg = {
.setkey = crc32c_vpmsum_setkey,
.init = crc32c_vpmsum_init,
.update = crc32c_vpmsum_update,
.final = crc32c_vpmsum_final,
.finup = crc32c_vpmsum_finup,
.digest = crc32c_vpmsum_digest,
.descsize = sizeof(u32),
.digestsize = CHKSUM_DIGEST_SIZE,
.base = {
.cra_name = "crc32c",
.cra_driver_name = "crc32c-vpmsum",
.cra_priority = 200,
.cra_blocksize = CHKSUM_BLOCK_SIZE,
.cra_ctxsize = sizeof(u32),
.cra_module = THIS_MODULE,
.cra_init = crc32c_vpmsum_cra_init,
}
};
static int __init crc32c_vpmsum_mod_init(void)
{
if (!cpu_has_feature(CPU_FTR_ARCH_207S))
return -ENODEV;
return crypto_register_shash(&alg);
}
static void __exit crc32c_vpmsum_mod_fini(void)
{
crypto_unregister_shash(&alg);
}
module_init(crc32c_vpmsum_mod_init);
module_exit(crc32c_vpmsum_mod_fini);
MODULE_AUTHOR("Anton Blanchard <anton@samba.org>");
MODULE_DESCRIPTION("CRC32C using vector polynomial multiply-sum instructions");
MODULE_LICENSE("GPL");
MODULE_ALIAS_CRYPTO("crc32c");
MODULE_ALIAS_CRYPTO("crc32c-vpmsum");
arch/powerpc/include/asm/ppc-opcode.h
浏览文件 @ bbce2ad2
......@@ -174,6 +174,8 @@
#define PPC_INST_MFSPR_DSCR_USER_MASK 0xfc1fffff
#define PPC_INST_MTSPR_DSCR_USER 0x7c0303a6
#define PPC_INST_MTSPR_DSCR_USER_MASK 0xfc1fffff
#define PPC_INST_MFVSRD 0x7c000066
#define PPC_INST_MTVSRD 0x7c000166
#define PPC_INST_SLBFEE 0x7c0007a7
#define PPC_INST_STRING 0x7c00042a
......@@ -188,6 +190,8 @@
#define PPC_INST_WAIT 0x7c00007c
#define PPC_INST_TLBIVAX 0x7c000624
#define PPC_INST_TLBSRX_DOT 0x7c0006a5
#define PPC_INST_VPMSUMW 0x10000488
#define PPC_INST_VPMSUMD 0x100004c8
#define PPC_INST_XXLOR 0xf0000510
#define PPC_INST_XXSWAPD 0xf0000250
#define PPC_INST_XVCPSGNDP 0xf0000780
......@@ -359,6 +363,14 @@
VSX_XX1((s), a, b))
#define LXVD2X(s, a, b) stringify_in_c(.long PPC_INST_LXVD2X | \
VSX_XX1((s), a, b))
#define MFVRD(a, t) stringify_in_c(.long PPC_INST_MFVSRD | \
VSX_XX1((t)+32, a, R0))
#define MTVRD(t, a) stringify_in_c(.long PPC_INST_MTVSRD | \
VSX_XX1((t)+32, a, R0))
#define VPMSUMW(t, a, b) stringify_in_c(.long PPC_INST_VPMSUMW | \
VSX_XX3((t), a, b))
#define VPMSUMD(t, a, b) stringify_in_c(.long PPC_INST_VPMSUMD | \
VSX_XX3((t), a, b))
#define XXLOR(t, a, b) stringify_in_c(.long PPC_INST_XXLOR | \
VSX_XX3((t), a, b))
#define XXSWAPD(t, a) stringify_in_c(.long PPC_INST_XXSWAPD | \
......
arch/powerpc/include/asm/ppc_asm.h
浏览文件 @ bbce2ad2
......@@ -286,6 +286,9 @@ GLUE(.,name):
#endif
#define FUNC_START(name) _GLOBAL(name)
#define FUNC_END(name)
/*
* LOAD_REG_IMMEDIATE(rn, expr)
* Loads the value of the constant expression 'expr' into register 'rn'
......
arch/powerpc/kernel/iomap.c
浏览文件 @ bbce2ad2
......@@ -38,6 +38,18 @@ EXPORT_SYMBOL(ioread16);
EXPORT_SYMBOL(ioread16be);
EXPORT_SYMBOL(ioread32);
EXPORT_SYMBOL(ioread32be);
#ifdef __powerpc64__
u64 ioread64(void __iomem *addr)
{
return readq(addr);
}
u64 ioread64be(void __iomem *addr)
{
return readq_be(addr);
}
EXPORT_SYMBOL(ioread64);
EXPORT_SYMBOL(ioread64be);
#endif /* __powerpc64__ */
void iowrite8(u8 val, void __iomem *addr)
{
......@@ -64,6 +76,18 @@ EXPORT_SYMBOL(iowrite16);
EXPORT_SYMBOL(iowrite16be);
EXPORT_SYMBOL(iowrite32);
EXPORT_SYMBOL(iowrite32be);
#ifdef __powerpc64__
void iowrite64(u64 val, void __iomem *addr)
{
writeq(val, addr);
}
void iowrite64be(u64 val, void __iomem *addr)
{
writeq_be(val, addr);
}
EXPORT_SYMBOL(iowrite64);
EXPORT_SYMBOL(iowrite64be);
#endif /* __powerpc64__ */
/*
* These are the "repeat read/write" functions. Note the
......
arch/s390/crypto/aes_s390.c
浏览文件 @ bbce2ad2
......@@ -22,6 +22,7 @@
#include <crypto/aes.h>
#include <crypto/algapi.h>
#include <crypto/internal/skcipher.h>
#include <linux/err.h>
#include <linux/module.h>
#include <linux/cpufeature.h>
......@@ -44,7 +45,7 @@ struct s390_aes_ctx {
long dec;
int key_len;
union {
struct crypto_blkcipher *blk;
struct crypto_skcipher *blk;
struct crypto_cipher *cip;
} fallback;
};
......@@ -63,7 +64,7 @@ struct s390_xts_ctx {
long enc;
long dec;
int key_len;
struct crypto_blkcipher *fallback;
struct crypto_skcipher *fallback;
};
/*
......@@ -237,16 +238,16 @@ static int setkey_fallback_blk(struct crypto_tfm *tfm, const u8 *key,
struct s390_aes_ctx *sctx = crypto_tfm_ctx(tfm);
unsigned int ret;
sctx->fallback.blk->base.crt_flags &= ~CRYPTO_TFM_REQ_MASK;
sctx->fallback.blk->base.crt_flags |= (tfm->crt_flags &
CRYPTO_TFM_REQ_MASK);
crypto_skcipher_clear_flags(sctx->fallback.blk, CRYPTO_TFM_REQ_MASK);
crypto_skcipher_set_flags(sctx->fallback.blk, tfm->crt_flags &
CRYPTO_TFM_REQ_MASK);
ret = crypto_skcipher_setkey(sctx->fallback.blk, key, len);
tfm->crt_flags &= ~CRYPTO_TFM_RES_MASK;
tfm->crt_flags |= crypto_skcipher_get_flags(sctx->fallback.blk) &
CRYPTO_TFM_RES_MASK;
ret = crypto_blkcipher_setkey(sctx->fallback.blk, key, len);
if (ret) {
tfm->crt_flags &= ~CRYPTO_TFM_RES_MASK;
tfm->crt_flags |= (sctx->fallback.blk->base.crt_flags &
CRYPTO_TFM_RES_MASK);
}
return ret;
}
......@@ -255,15 +256,17 @@ static int fallback_blk_dec(struct blkcipher_desc *desc,
unsigned int nbytes)
{
unsigned int ret;
struct crypto_blkcipher *tfm;
struct s390_aes_ctx *sctx = crypto_blkcipher_ctx(desc->tfm);
struct crypto_blkcipher *tfm = desc->tfm;
struct s390_aes_ctx *sctx = crypto_blkcipher_ctx(tfm);
SKCIPHER_REQUEST_ON_STACK(req, sctx->fallback.blk);
tfm = desc->tfm;
desc->tfm = sctx->fallback.blk;
skcipher_request_set_tfm(req, sctx->fallback.blk);
skcipher_request_set_callback(req, desc->flags, NULL, NULL);
skcipher_request_set_crypt(req, src, dst, nbytes, desc->info);
ret = crypto_blkcipher_decrypt_iv(desc, dst, src, nbytes);
ret = crypto_skcipher_decrypt(req);
desc->tfm = tfm;
skcipher_request_zero(req);
return ret;
}
......@@ -272,15 +275,15 @@ static int fallback_blk_enc(struct blkcipher_desc *desc,
unsigned int nbytes)
{
unsigned int ret;
struct crypto_blkcipher *tfm;
struct s390_aes_ctx *sctx = crypto_blkcipher_ctx(desc->tfm);
struct crypto_blkcipher *tfm = desc->tfm;
struct s390_aes_ctx *sctx = crypto_blkcipher_ctx(tfm);
SKCIPHER_REQUEST_ON_STACK(req, sctx->fallback.blk);
tfm = desc->tfm;
desc->tfm = sctx->fallback.blk;
skcipher_request_set_tfm(req, sctx->fallback.blk);
skcipher_request_set_callback(req, desc->flags, NULL, NULL);
skcipher_request_set_crypt(req, src, dst, nbytes, desc->info);
ret = crypto_blkcipher_encrypt_iv(desc, dst, src, nbytes);
desc->tfm = tfm;
ret = crypto_skcipher_encrypt(req);
return ret;
}
......@@ -370,8 +373,9 @@ static int fallback_init_blk(struct crypto_tfm *tfm)
const char *name = tfm->__crt_alg->cra_name;
struct s390_aes_ctx *sctx = crypto_tfm_ctx(tfm);
sctx->fallback.blk = crypto_alloc_blkcipher(name, 0,
CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK);
sctx->fallback.blk = crypto_alloc_skcipher(name, 0,
CRYPTO_ALG_ASYNC |
CRYPTO_ALG_NEED_FALLBACK);
if (IS_ERR(sctx->fallback.blk)) {
pr_err("Allocating AES fallback algorithm %s failed\n",
......@@ -386,8 +390,7 @@ static void fallback_exit_blk(struct crypto_tfm *tfm)
{
struct s390_aes_ctx *sctx = crypto_tfm_ctx(tfm);
crypto_free_blkcipher(sctx->fallback.blk);
sctx->fallback.blk = NULL;
crypto_free_skcipher(sctx->fallback.blk);
}
static struct crypto_alg ecb_aes_alg = {
......@@ -536,16 +539,16 @@ static int xts_fallback_setkey(struct crypto_tfm *tfm, const u8 *key,
struct s390_xts_ctx *xts_ctx = crypto_tfm_ctx(tfm);
unsigned int ret;
xts_ctx->fallback->base.crt_flags &= ~CRYPTO_TFM_REQ_MASK;
xts_ctx->fallback->base.crt_flags |= (tfm->crt_flags &
CRYPTO_TFM_REQ_MASK);
crypto_skcipher_clear_flags(xts_ctx->fallback, CRYPTO_TFM_REQ_MASK);
crypto_skcipher_set_flags(xts_ctx->fallback, tfm->crt_flags &
CRYPTO_TFM_REQ_MASK);
ret = crypto_skcipher_setkey(xts_ctx->fallback, key, len);
tfm->crt_flags &= ~CRYPTO_TFM_RES_MASK;
tfm->crt_flags |= crypto_skcipher_get_flags(xts_ctx->fallback) &
CRYPTO_TFM_RES_MASK;
ret = crypto_blkcipher_setkey(xts_ctx->fallback, key, len);
if (ret) {
tfm->crt_flags &= ~CRYPTO_TFM_RES_MASK;
tfm->crt_flags |= (xts_ctx->fallback->base.crt_flags &
CRYPTO_TFM_RES_MASK);
}
return ret;
}
......@@ -553,16 +556,18 @@ static int xts_fallback_decrypt(struct blkcipher_desc *desc,
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
{
struct s390_xts_ctx *xts_ctx = crypto_blkcipher_ctx(desc->tfm);
struct crypto_blkcipher *tfm;
struct crypto_blkcipher *tfm = desc->tfm;
struct s390_xts_ctx *xts_ctx = crypto_blkcipher_ctx(tfm);
SKCIPHER_REQUEST_ON_STACK(req, xts_ctx->fallback);
unsigned int ret;
tfm = desc->tfm;
desc->tfm = xts_ctx->fallback;
skcipher_request_set_tfm(req, xts_ctx->fallback);
skcipher_request_set_callback(req, desc->flags, NULL, NULL);
skcipher_request_set_crypt(req, src, dst, nbytes, desc->info);
ret = crypto_blkcipher_decrypt_iv(desc, dst, src, nbytes);
ret = crypto_skcipher_decrypt(req);
desc->tfm = tfm;
skcipher_request_zero(req);
return ret;
}
......@@ -570,16 +575,18 @@ static int xts_fallback_encrypt(struct blkcipher_desc *desc,
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
{
struct s390_xts_ctx *xts_ctx = crypto_blkcipher_ctx(desc->tfm);
struct crypto_blkcipher *tfm;
struct crypto_blkcipher *tfm = desc->tfm;
struct s390_xts_ctx *xts_ctx = crypto_blkcipher_ctx(tfm);
SKCIPHER_REQUEST_ON_STACK(req, xts_ctx->fallback);
unsigned int ret;
tfm = desc->tfm;
desc->tfm = xts_ctx->fallback;
skcipher_request_set_tfm(req, xts_ctx->fallback);
skcipher_request_set_callback(req, desc->flags, NULL, NULL);
skcipher_request_set_crypt(req, src, dst, nbytes, desc->info);
ret = crypto_blkcipher_encrypt_iv(desc, dst, src, nbytes);
ret = crypto_skcipher_encrypt(req);
desc->tfm = tfm;
skcipher_request_zero(req);
return ret;
}
......@@ -700,8 +707,9 @@ static int xts_fallback_init(struct crypto_tfm *tfm)
const char *name = tfm->__crt_alg->cra_name;
struct s390_xts_ctx *xts_ctx = crypto_tfm_ctx(tfm);
xts_ctx->fallback = crypto_alloc_blkcipher(name, 0,
CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK);
xts_ctx->fallback = crypto_alloc_skcipher(name, 0,
CRYPTO_ALG_ASYNC |
CRYPTO_ALG_NEED_FALLBACK);
if (IS_ERR(xts_ctx->fallback)) {
pr_err("Allocating XTS fallback algorithm %s failed\n",
......@@ -715,8 +723,7 @@ static void xts_fallback_exit(struct crypto_tfm *tfm)
{
struct s390_xts_ctx *xts_ctx = crypto_tfm_ctx(tfm);
crypto_free_blkcipher(xts_ctx->fallback);
xts_ctx->fallback = NULL;
crypto_free_skcipher(xts_ctx->fallback);
}
static struct crypto_alg xts_aes_alg = {
......
arch/x86/crypto/Makefile
浏览文件 @ bbce2ad2
......@@ -49,7 +49,9 @@ endif
ifeq ($(avx2_supported),yes)
obj-$(CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64) += camellia-aesni-avx2.o
obj-$(CONFIG_CRYPTO_SERPENT_AVX2_X86_64) += serpent-avx2.o
obj-$(CONFIG_CRYPTO_SHA1_MB) += sha-mb/
obj-$(CONFIG_CRYPTO_SHA1_MB) += sha1-mb/
obj-$(CONFIG_CRYPTO_SHA256_MB) += sha256-mb/
obj-$(CONFIG_CRYPTO_SHA512_MB) += sha512-mb/
endif
aes-i586-y := aes-i586-asm_32.o aes_glue.o
......
arch/x86/crypto/aesni-intel_glue.c
浏览文件 @ bbce2ad2
......@@ -59,17 +59,6 @@ struct aesni_rfc4106_gcm_ctx {
u8 nonce[4];
};
struct aesni_gcm_set_hash_subkey_result {
int err;
struct completion completion;
};
struct aesni_hash_subkey_req_data {
u8 iv[16];
struct aesni_gcm_set_hash_subkey_result result;
struct scatterlist sg;
};
struct aesni_lrw_ctx {
struct lrw_table_ctx lrw_table;
u8 raw_aes_ctx[sizeof(struct crypto_aes_ctx) + AESNI_ALIGN - 1];
......@@ -809,71 +798,28 @@ static void rfc4106_exit(struct crypto_aead *aead)
cryptd_free_aead(*ctx);
}
static void
rfc4106_set_hash_subkey_done(struct crypto_async_request *req, int err)
{
struct aesni_gcm_set_hash_subkey_result *result = req->data;
if (err == -EINPROGRESS)
return;
result->err = err;
complete(&result->completion);
}
static int
rfc4106_set_hash_subkey(u8 *hash_subkey, const u8 *key, unsigned int key_len)
{
struct crypto_ablkcipher *ctr_tfm;
struct ablkcipher_request *req;
int ret = -EINVAL;
struct aesni_hash_subkey_req_data *req_data;
struct crypto_cipher *tfm;
int ret;
ctr_tfm = crypto_alloc_ablkcipher("ctr(aes)", 0, 0);
if (IS_ERR(ctr_tfm))
return PTR_ERR(ctr_tfm);
tfm = crypto_alloc_cipher("aes", 0, 0);
if (IS_ERR(tfm))
return PTR_ERR(tfm);
ret = crypto_ablkcipher_setkey(ctr_tfm, key, key_len);
ret = crypto_cipher_setkey(tfm, key, key_len);
if (ret)
goto out_free_ablkcipher;
ret = -ENOMEM;
req = ablkcipher_request_alloc(ctr_tfm, GFP_KERNEL);
if (!req)
goto out_free_ablkcipher;
req_data = kmalloc(sizeof(*req_data), GFP_KERNEL);
if (!req_data)
goto out_free_request;
memset(req_data->iv, 0, sizeof(req_data->iv));
goto out_free_cipher;
/* Clear the data in the hash sub key container to zero.*/
/* We want to cipher all zeros to create the hash sub key. */
memset(hash_subkey, 0, RFC4106_HASH_SUBKEY_SIZE);
init_completion(&req_data->result.completion);
sg_init_one(&req_data->sg, hash_subkey, RFC4106_HASH_SUBKEY_SIZE);
ablkcipher_request_set_tfm(req, ctr_tfm);
ablkcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP |
CRYPTO_TFM_REQ_MAY_BACKLOG,
rfc4106_set_hash_subkey_done,
&req_data->result);
ablkcipher_request_set_crypt(req, &req_data->sg,
&req_data->sg, RFC4106_HASH_SUBKEY_SIZE, req_data->iv);
ret = crypto_ablkcipher_encrypt(req);
if (ret == -EINPROGRESS || ret == -EBUSY) {
ret = wait_for_completion_interruptible
(&req_data->result.completion);
if (!ret)
ret = req_data->result.err;
}
kfree(req_data);
out_free_request:
ablkcipher_request_free(req);
out_free_ablkcipher:
crypto_free_ablkcipher(ctr_tfm);
crypto_cipher_encrypt_one(tfm, hash_subkey, hash_subkey);
out_free_cipher:
crypto_free_cipher(tfm);
return ret;
}
......@@ -1098,9 +1044,12 @@ static int rfc4106_encrypt(struct aead_request *req)
struct cryptd_aead **ctx = crypto_aead_ctx(tfm);
struct cryptd_aead *cryptd_tfm = *ctx;
aead_request_set_tfm(req, irq_fpu_usable() ?
cryptd_aead_child(cryptd_tfm) :
&cryptd_tfm->base);
tfm = &cryptd_tfm->base;
if (irq_fpu_usable() && (!in_atomic() ||
!cryptd_aead_queued(cryptd_tfm)))
tfm = cryptd_aead_child(cryptd_tfm);
aead_request_set_tfm(req, tfm);
return crypto_aead_encrypt(req);
}
......@@ -1111,9 +1060,12 @@ static int rfc4106_decrypt(struct aead_request *req)
struct cryptd_aead **ctx = crypto_aead_ctx(tfm);
struct cryptd_aead *cryptd_tfm = *ctx;
aead_request_set_tfm(req, irq_fpu_usable() ?
cryptd_aead_child(cryptd_tfm) :
&cryptd_tfm->base);
tfm = &cryptd_tfm->base;
if (irq_fpu_usable() && (!in_atomic() ||
!cryptd_aead_queued(cryptd_tfm)))
tfm = cryptd_aead_child(cryptd_tfm);
aead_request_set_tfm(req, tfm);
return crypto_aead_decrypt(req);
}
......
arch/x86/crypto/chacha20_glue.c
浏览文件 @ bbce2ad2
......@@ -70,7 +70,7 @@ static int chacha20_simd(struct blkcipher_desc *desc, struct scatterlist *dst,
struct blkcipher_walk walk;
int err;
if (!may_use_simd())
if (nbytes <= CHACHA20_BLOCK_SIZE || !may_use_simd())
return crypto_chacha20_crypt(desc, dst, src, nbytes);
state = (u32 *)roundup((uintptr_t)state_buf, CHACHA20_STATE_ALIGN);
......
arch/x86/crypto/ghash-clmulni-intel_glue.c
浏览文件 @ bbce2ad2
......@@ -168,30 +168,23 @@ static int ghash_async_init(struct ahash_request *req)
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
struct crypto_shash *child = cryptd_ahash_child(cryptd_tfm);
if (!irq_fpu_usable()) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_init(cryptd_req);
} else {
struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
struct crypto_shash *child = cryptd_ahash_child(cryptd_tfm);
desc->tfm = child;
desc->flags = req->base.flags;
return crypto_shash_init(desc);
}
desc->tfm = child;
desc->flags = req->base.flags;
return crypto_shash_init(desc);
}
static int ghash_async_update(struct ahash_request *req)
{
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
if (!irq_fpu_usable()) {
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
if (!irq_fpu_usable() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_update(cryptd_req);
......@@ -204,12 +197,12 @@ static int ghash_async_update(struct ahash_request *req)
static int ghash_async_final(struct ahash_request *req)
{
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
if (!irq_fpu_usable()) {
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
if (!irq_fpu_usable() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_final(cryptd_req);
......@@ -249,7 +242,8 @@ static int ghash_async_digest(struct ahash_request *req)
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
if (!irq_fpu_usable()) {
if (!irq_fpu_usable() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_digest(cryptd_req);
......
arch/x86/crypto/sha-mb/sha1_mb.c → arch/x86/crypto/sha1-mb/sha1_mb.c
浏览文件 @ bbce2ad2
......@@ -67,7 +67,7 @@
#include <asm/byteorder.h>
#include <linux/hardirq.h>
#include <asm/fpu/api.h>
#include "sha_mb_ctx.h"
#include "sha1_mb_ctx.h"
#define FLUSH_INTERVAL 1000 /* in usec */
......@@ -77,30 +77,34 @@ struct sha1_mb_ctx {
struct mcryptd_ahash *mcryptd_tfm;
};
static inline struct mcryptd_hash_request_ctx *cast_hash_to_mcryptd_ctx(struct sha1_hash_ctx *hash_ctx)
static inline struct mcryptd_hash_request_ctx
*cast_hash_to_mcryptd_ctx(struct sha1_hash_ctx *hash_ctx)
{
struct shash_desc *desc;
struct ahash_request *areq;
desc = container_of((void *) hash_ctx, struct shash_desc, __ctx);
return container_of(desc, struct mcryptd_hash_request_ctx, desc);
areq = container_of((void *) hash_ctx, struct ahash_request, __ctx);
return container_of(areq, struct mcryptd_hash_request_ctx, areq);
}
static inline struct ahash_request *cast_mcryptd_ctx_to_req(struct mcryptd_hash_request_ctx *ctx)
static inline struct ahash_request
*cast_mcryptd_ctx_to_req(struct mcryptd_hash_request_ctx *ctx)
{
return container_of((void *) ctx, struct ahash_request, __ctx);
}
static void req_ctx_init(struct mcryptd_hash_request_ctx *rctx,
struct shash_desc *desc)
struct ahash_request *areq)
{
rctx->flag = HASH_UPDATE;
}
static asmlinkage void (*sha1_job_mgr_init)(struct sha1_mb_mgr *state);
static asmlinkage struct job_sha1* (*sha1_job_mgr_submit)(struct sha1_mb_mgr *state,
struct job_sha1 *job);
static asmlinkage struct job_sha1* (*sha1_job_mgr_flush)(struct sha1_mb_mgr *state);
static asmlinkage struct job_sha1* (*sha1_job_mgr_get_comp_job)(struct sha1_mb_mgr *state);
static asmlinkage struct job_sha1* (*sha1_job_mgr_submit)
(struct sha1_mb_mgr *state, struct job_sha1 *job);
static asmlinkage struct job_sha1* (*sha1_job_mgr_flush)
(struct sha1_mb_mgr *state);
static asmlinkage struct job_sha1* (*sha1_job_mgr_get_comp_job)
(struct sha1_mb_mgr *state);
static inline void sha1_init_digest(uint32_t *digest)
{
......@@ -131,7 +135,8 @@ static inline uint32_t sha1_pad(uint8_t padblock[SHA1_BLOCK_SIZE * 2],
return i >> SHA1_LOG2_BLOCK_SIZE;
}
static struct sha1_hash_ctx *sha1_ctx_mgr_resubmit(struct sha1_ctx_mgr *mgr, struct sha1_hash_ctx *ctx)
static struct sha1_hash_ctx *sha1_ctx_mgr_resubmit(struct sha1_ctx_mgr *mgr,
struct sha1_hash_ctx *ctx)
{
while (ctx) {
if (ctx->status & HASH_CTX_STS_COMPLETE) {
......@@ -177,8 +182,8 @@ static struct sha1_hash_ctx *sha1_ctx_mgr_resubmit(struct sha1_ctx_mgr *mgr, str
ctx->job.buffer = (uint8_t *) buffer;
ctx->job.len = len;
ctx = (struct sha1_hash_ctx *) sha1_job_mgr_submit(&mgr->mgr,
&ctx->job);
ctx = (struct sha1_hash_ctx *)sha1_job_mgr_submit(&mgr->mgr,
&ctx->job);
continue;
}
}
......@@ -191,13 +196,15 @@ static struct sha1_hash_ctx *sha1_ctx_mgr_resubmit(struct sha1_ctx_mgr *mgr, str
if (ctx->status & HASH_CTX_STS_LAST) {
uint8_t *buf = ctx->partial_block_buffer;
uint32_t n_extra_blocks = sha1_pad(buf, ctx->total_length);
uint32_t n_extra_blocks =
sha1_pad(buf, ctx->total_length);
ctx->status = (HASH_CTX_STS_PROCESSING |
HASH_CTX_STS_COMPLETE);
ctx->job.buffer = buf;
ctx->job.len = (uint32_t) n_extra_blocks;
ctx = (struct sha1_hash_ctx *) sha1_job_mgr_submit(&mgr->mgr, &ctx->job);
ctx = (struct sha1_hash_ctx *)
sha1_job_mgr_submit(&mgr->mgr, &ctx->job);
continue;
}
......@@ -208,14 +215,17 @@ static struct sha1_hash_ctx *sha1_ctx_mgr_resubmit(struct sha1_ctx_mgr *mgr, str
return NULL;
}
static struct sha1_hash_ctx *sha1_ctx_mgr_get_comp_ctx(struct sha1_ctx_mgr *mgr)
static struct sha1_hash_ctx
*sha1_ctx_mgr_get_comp_ctx(struct sha1_ctx_mgr *mgr)
{
/*
* If get_comp_job returns NULL, there are no jobs complete.
* If get_comp_job returns a job, verify that it is safe to return to the user.
* If get_comp_job returns a job, verify that it is safe to return to
* the user.
* If it is not ready, resubmit the job to finish processing.
* If sha1_ctx_mgr_resubmit returned a job, it is ready to be returned.
* Otherwise, all jobs currently being managed by the hash_ctx_mgr still need processing.
* Otherwise, all jobs currently being managed by the hash_ctx_mgr
* still need processing.
*/
struct sha1_hash_ctx *ctx;
......@@ -235,7 +245,10 @@ static struct sha1_hash_ctx *sha1_ctx_mgr_submit(struct sha1_ctx_mgr *mgr,
int flags)
{
if (flags & (~HASH_ENTIRE)) {
/* User should not pass anything other than FIRST, UPDATE, or LAST */
/*
* User should not pass anything other than FIRST, UPDATE, or
* LAST
*/
ctx->error = HASH_CTX_ERROR_INVALID_FLAGS;
return ctx;
}
......@@ -264,14 +277,20 @@ static struct sha1_hash_ctx *sha1_ctx_mgr_submit(struct sha1_ctx_mgr *mgr,
ctx->partial_block_buffer_length = 0;
}
/* If we made it here, there were no errors during this call to submit */
/*
* If we made it here, there were no errors during this call to
* submit
*/
ctx->error = HASH_CTX_ERROR_NONE;
/* Store buffer ptr info from user */
ctx->incoming_buffer = buffer;
ctx->incoming_buffer_length = len;
/* Store the user's request flags and mark this ctx as currently being processed. */
/*
* Store the user's request flags and mark this ctx as currently
* being processed.
*/
ctx->status = (flags & HASH_LAST) ?
(HASH_CTX_STS_PROCESSING | HASH_CTX_STS_LAST) :
HASH_CTX_STS_PROCESSING;
......@@ -285,9 +304,13 @@ static struct sha1_hash_ctx *sha1_ctx_mgr_submit(struct sha1_ctx_mgr *mgr,
* Or if the user's buffer contains less than a whole block,
* append as much as possible to the extra block.
*/
if ((ctx->partial_block_buffer_length) | (len < SHA1_BLOCK_SIZE)) {
/* Compute how many bytes to copy from user buffer into extra block */
uint32_t copy_len = SHA1_BLOCK_SIZE - ctx->partial_block_buffer_length;
if (ctx->partial_block_buffer_length || len < SHA1_BLOCK_SIZE) {
/*
* Compute how many bytes to copy from user buffer into
* extra block
*/
uint32_t copy_len = SHA1_BLOCK_SIZE -
ctx->partial_block_buffer_length;
if (len < copy_len)
copy_len = len;
......@@ -297,20 +320,28 @@ static struct sha1_hash_ctx *sha1_ctx_mgr_submit(struct sha1_ctx_mgr *mgr,
buffer, copy_len);
ctx->partial_block_buffer_length += copy_len;
ctx->incoming_buffer = (const void *)((const char *)buffer + copy_len);
ctx->incoming_buffer = (const void *)
((const char *)buffer + copy_len);
ctx->incoming_buffer_length = len - copy_len;
}
/* The extra block should never contain more than 1 block here */
/*
* The extra block should never contain more than 1 block
* here
*/
assert(ctx->partial_block_buffer_length <= SHA1_BLOCK_SIZE);
/* If the extra block buffer contains exactly 1 block, it can be hashed. */
/*
* If the extra block buffer contains exactly 1 block, it can
* be hashed.
*/
if (ctx->partial_block_buffer_length >= SHA1_BLOCK_SIZE) {
ctx->partial_block_buffer_length = 0;
ctx->job.buffer = ctx->partial_block_buffer;
ctx->job.len = 1;
ctx = (struct sha1_hash_ctx *) sha1_job_mgr_submit(&mgr->mgr, &ctx->job);
ctx = (struct sha1_hash_ctx *)
sha1_job_mgr_submit(&mgr->mgr, &ctx->job);
}
}
......@@ -329,23 +360,24 @@ static struct sha1_hash_ctx *sha1_ctx_mgr_flush(struct sha1_ctx_mgr *mgr)
return NULL;
/*
* If flush returned a job, resubmit the job to finish processing.
* If flush returned a job, resubmit the job to finish
* processing.
*/
ctx = sha1_ctx_mgr_resubmit(mgr, ctx);
/*
* If sha1_ctx_mgr_resubmit returned a job, it is ready to be returned.
* Otherwise, all jobs currently being managed by the sha1_ctx_mgr
* still need processing. Loop.
* If sha1_ctx_mgr_resubmit returned a job, it is ready to be
* returned. Otherwise, all jobs currently being managed by the
* sha1_ctx_mgr still need processing. Loop.
*/
if (ctx)
return ctx;
}
}
static int sha1_mb_init(struct shash_desc *desc)
static int sha1_mb_init(struct ahash_request *areq)
{
struct sha1_hash_ctx *sctx = shash_desc_ctx(desc);
struct sha1_hash_ctx *sctx = ahash_request_ctx(areq);
hash_ctx_init(sctx);
sctx->job.result_digest[0] = SHA1_H0;
......@@ -363,7 +395,7 @@ static int sha1_mb_init(struct shash_desc *desc)
static int sha1_mb_set_results(struct mcryptd_hash_request_ctx *rctx)
{
int i;
struct sha1_hash_ctx *sctx = shash_desc_ctx(&rctx->desc);
struct sha1_hash_ctx *sctx = ahash_request_ctx(&rctx->areq);
__be32 *dst = (__be32 *) rctx->out;
for (i = 0; i < 5; ++i)
......@@ -394,9 +426,11 @@ static int sha_finish_walk(struct mcryptd_hash_request_ctx **ret_rctx,
flag |= HASH_LAST;
}
sha_ctx = (struct sha1_hash_ctx *) shash_desc_ctx(&rctx->desc);
sha_ctx = (struct sha1_hash_ctx *)
ahash_request_ctx(&rctx->areq);
kernel_fpu_begin();
sha_ctx = sha1_ctx_mgr_submit(cstate->mgr, sha_ctx, rctx->walk.data, nbytes, flag);
sha_ctx = sha1_ctx_mgr_submit(cstate->mgr, sha_ctx,
rctx->walk.data, nbytes, flag);
if (!sha_ctx) {
if (flush)
sha_ctx = sha1_ctx_mgr_flush(cstate->mgr);
......@@ -485,11 +519,10 @@ static void sha1_mb_add_list(struct mcryptd_hash_request_ctx *rctx,
mcryptd_arm_flusher(cstate, delay);
}
static int sha1_mb_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
static int sha1_mb_update(struct ahash_request *areq)
{
struct mcryptd_hash_request_ctx *rctx =
container_of(desc, struct mcryptd_hash_request_ctx, desc);
container_of(areq, struct mcryptd_hash_request_ctx, areq);
struct mcryptd_alg_cstate *cstate =
this_cpu_ptr(sha1_mb_alg_state.alg_cstate);
......@@ -505,7 +538,7 @@ static int sha1_mb_update(struct shash_desc *desc, const u8 *data,
}
/* need to init context */
req_ctx_init(rctx, desc);
req_ctx_init(rctx, areq);
nbytes = crypto_ahash_walk_first(req, &rctx->walk);
......@@ -518,10 +551,11 @@ static int sha1_mb_update(struct shash_desc *desc, const u8 *data,
rctx->flag |= HASH_DONE;
/* submit */
sha_ctx = (struct sha1_hash_ctx *) shash_desc_ctx(desc);
sha_ctx = (struct sha1_hash_ctx *) ahash_request_ctx(areq);
sha1_mb_add_list(rctx, cstate);
kernel_fpu_begin();
sha_ctx = sha1_ctx_mgr_submit(cstate->mgr, sha_ctx, rctx->walk.data, nbytes, HASH_UPDATE);
sha_ctx = sha1_ctx_mgr_submit(cstate->mgr, sha_ctx, rctx->walk.data,
nbytes, HASH_UPDATE);
kernel_fpu_end();
/* check if anything is returned */
......@@ -544,11 +578,10 @@ static int sha1_mb_update(struct shash_desc *desc, const u8 *data,
return ret;
}
static int sha1_mb_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
static int sha1_mb_finup(struct ahash_request *areq)
{
struct mcryptd_hash_request_ctx *rctx =
container_of(desc, struct mcryptd_hash_request_ctx, desc);
container_of(areq, struct mcryptd_hash_request_ctx, areq);
struct mcryptd_alg_cstate *cstate =
this_cpu_ptr(sha1_mb_alg_state.alg_cstate);
......@@ -563,7 +596,7 @@ static int sha1_mb_finup(struct shash_desc *desc, const u8 *data,
}
/* need to init context */
req_ctx_init(rctx, desc);
req_ctx_init(rctx, areq);
nbytes = crypto_ahash_walk_first(req, &rctx->walk);
......@@ -576,15 +609,15 @@ static int sha1_mb_finup(struct shash_desc *desc, const u8 *data,
rctx->flag |= HASH_DONE;
flag = HASH_LAST;
}
rctx->out = out;
/* submit */
rctx->flag |= HASH_FINAL;
sha_ctx = (struct sha1_hash_ctx *) shash_desc_ctx(desc);
sha_ctx = (struct sha1_hash_ctx *) ahash_request_ctx(areq);
sha1_mb_add_list(rctx, cstate);
kernel_fpu_begin();
sha_ctx = sha1_ctx_mgr_submit(cstate->mgr, sha_ctx, rctx->walk.data, nbytes, flag);
sha_ctx = sha1_ctx_mgr_submit(cstate->mgr, sha_ctx, rctx->walk.data,
nbytes, flag);
kernel_fpu_end();
/* check if anything is returned */
......@@ -605,10 +638,10 @@ static int sha1_mb_finup(struct shash_desc *desc, const u8 *data,
return ret;
}
static int sha1_mb_final(struct shash_desc *desc, u8 *out)
static int sha1_mb_final(struct ahash_request *areq)
{
struct mcryptd_hash_request_ctx *rctx =
container_of(desc, struct mcryptd_hash_request_ctx, desc);
container_of(areq, struct mcryptd_hash_request_ctx, areq);
struct mcryptd_alg_cstate *cstate =
this_cpu_ptr(sha1_mb_alg_state.alg_cstate);
......@@ -623,16 +656,16 @@ static int sha1_mb_final(struct shash_desc *desc, u8 *out)
}
/* need to init context */
req_ctx_init(rctx, desc);
req_ctx_init(rctx, areq);
rctx->out = out;
rctx->flag |= HASH_DONE | HASH_FINAL;
sha_ctx = (struct sha1_hash_ctx *) shash_desc_ctx(desc);
sha_ctx = (struct sha1_hash_ctx *) ahash_request_ctx(areq);
/* flag HASH_FINAL and 0 data size */
sha1_mb_add_list(rctx, cstate);
kernel_fpu_begin();
sha_ctx = sha1_ctx_mgr_submit(cstate->mgr, sha_ctx, &data, 0, HASH_LAST);
sha_ctx = sha1_ctx_mgr_submit(cstate->mgr, sha_ctx, &data, 0,
HASH_LAST);
kernel_fpu_end();
/* check if anything is returned */
......@@ -654,48 +687,98 @@ static int sha1_mb_final(struct shash_desc *desc, u8 *out)
return ret;
}
static int sha1_mb_export(struct shash_desc *desc, void *out)
static int sha1_mb_export(struct ahash_request *areq, void *out)
{
struct sha1_hash_ctx *sctx = shash_desc_ctx(desc);
struct sha1_hash_ctx *sctx = ahash_request_ctx(areq);
memcpy(out, sctx, sizeof(*sctx));
return 0;
}
static int sha1_mb_import(struct shash_desc *desc, const void *in)
static int sha1_mb_import(struct ahash_request *areq, const void *in)
{
struct sha1_hash_ctx *sctx = shash_desc_ctx(desc);
struct sha1_hash_ctx *sctx = ahash_request_ctx(areq);
memcpy(sctx, in, sizeof(*sctx));
return 0;
}
static int sha1_mb_async_init_tfm(struct crypto_tfm *tfm)
{
struct mcryptd_ahash *mcryptd_tfm;
struct sha1_mb_ctx *ctx = crypto_tfm_ctx(tfm);
struct mcryptd_hash_ctx *mctx;
static struct shash_alg sha1_mb_shash_alg = {
.digestsize = SHA1_DIGEST_SIZE,
mcryptd_tfm = mcryptd_alloc_ahash("__intel_sha1-mb",
CRYPTO_ALG_INTERNAL,
CRYPTO_ALG_INTERNAL);
if (IS_ERR(mcryptd_tfm))
return PTR_ERR(mcryptd_tfm);
mctx = crypto_ahash_ctx(&mcryptd_tfm->base);
mctx->alg_state = &sha1_mb_alg_state;
ctx->mcryptd_tfm = mcryptd_tfm;
crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm),
sizeof(struct ahash_request) +
crypto_ahash_reqsize(&mcryptd_tfm->base));
return 0;
}
static void sha1_mb_async_exit_tfm(struct crypto_tfm *tfm)
{
struct sha1_mb_ctx *ctx = crypto_tfm_ctx(tfm);
mcryptd_free_ahash(ctx->mcryptd_tfm);
}
static int sha1_mb_areq_init_tfm(struct crypto_tfm *tfm)
{
crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm),
sizeof(struct ahash_request) +
sizeof(struct sha1_hash_ctx));
return 0;
}
static void sha1_mb_areq_exit_tfm(struct crypto_tfm *tfm)
{
struct sha1_mb_ctx *ctx = crypto_tfm_ctx(tfm);
mcryptd_free_ahash(ctx->mcryptd_tfm);
}
static struct ahash_alg sha1_mb_areq_alg = {
.init = sha1_mb_init,
.update = sha1_mb_update,
.final = sha1_mb_final,
.finup = sha1_mb_finup,
.export = sha1_mb_export,
.import = sha1_mb_import,
.descsize = sizeof(struct sha1_hash_ctx),
.statesize = sizeof(struct sha1_hash_ctx),
.base = {
.cra_name = "__sha1-mb",
.cra_driver_name = "__intel_sha1-mb",
.cra_priority = 100,
/*
* use ASYNC flag as some buffers in multi-buffer
* algo may not have completed before hashing thread sleep
*/
.cra_flags = CRYPTO_ALG_TYPE_SHASH | CRYPTO_ALG_ASYNC |
CRYPTO_ALG_INTERNAL,
.cra_blocksize = SHA1_BLOCK_SIZE,
.cra_module = THIS_MODULE,
.cra_list = LIST_HEAD_INIT(sha1_mb_shash_alg.base.cra_list),
.halg = {
.digestsize = SHA1_DIGEST_SIZE,
.statesize = sizeof(struct sha1_hash_ctx),
.base = {
.cra_name = "__sha1-mb",
.cra_driver_name = "__intel_sha1-mb",
.cra_priority = 100,
/*
* use ASYNC flag as some buffers in multi-buffer
* algo may not have completed before hashing thread
* sleep
*/
.cra_flags = CRYPTO_ALG_TYPE_AHASH |
CRYPTO_ALG_ASYNC |
CRYPTO_ALG_INTERNAL,
.cra_blocksize = SHA1_BLOCK_SIZE,
.cra_module = THIS_MODULE,
.cra_list = LIST_HEAD_INIT
(sha1_mb_areq_alg.halg.base.cra_list),
.cra_init = sha1_mb_areq_init_tfm,
.cra_exit = sha1_mb_areq_exit_tfm,
.cra_ctxsize = sizeof(struct sha1_hash_ctx),
}
}
};
......@@ -780,46 +863,20 @@ static int sha1_mb_async_import(struct ahash_request *req, const void *in)
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
struct sha1_mb_ctx *ctx = crypto_ahash_ctx(tfm);
struct mcryptd_ahash *mcryptd_tfm = ctx->mcryptd_tfm;
struct crypto_shash *child = mcryptd_ahash_child(mcryptd_tfm);
struct crypto_ahash *child = mcryptd_ahash_child(mcryptd_tfm);
struct mcryptd_hash_request_ctx *rctx;
struct shash_desc *desc;
struct ahash_request *areq;
memcpy(mcryptd_req, req, sizeof(*req));
ahash_request_set_tfm(mcryptd_req, &mcryptd_tfm->base);
rctx = ahash_request_ctx(mcryptd_req);
desc = &rctx->desc;
desc->tfm = child;
desc->flags = CRYPTO_TFM_REQ_MAY_SLEEP;
return crypto_ahash_import(mcryptd_req, in);
}
static int sha1_mb_async_init_tfm(struct crypto_tfm *tfm)
{
struct mcryptd_ahash *mcryptd_tfm;
struct sha1_mb_ctx *ctx = crypto_tfm_ctx(tfm);
struct mcryptd_hash_ctx *mctx;
areq = &rctx->areq;
mcryptd_tfm = mcryptd_alloc_ahash("__intel_sha1-mb",
CRYPTO_ALG_INTERNAL,
CRYPTO_ALG_INTERNAL);
if (IS_ERR(mcryptd_tfm))
return PTR_ERR(mcryptd_tfm);
mctx = crypto_ahash_ctx(&mcryptd_tfm->base);
mctx->alg_state = &sha1_mb_alg_state;
ctx->mcryptd_tfm = mcryptd_tfm;
crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm),
sizeof(struct ahash_request) +
crypto_ahash_reqsize(&mcryptd_tfm->base));
ahash_request_set_tfm(areq, child);
ahash_request_set_callback(areq, CRYPTO_TFM_REQ_MAY_SLEEP,
rctx->complete, req);
return 0;
}
static void sha1_mb_async_exit_tfm(struct crypto_tfm *tfm)
{
struct sha1_mb_ctx *ctx = crypto_tfm_ctx(tfm);
mcryptd_free_ahash(ctx->mcryptd_tfm);
return crypto_ahash_import(mcryptd_req, in);
}
static struct ahash_alg sha1_mb_async_alg = {
......@@ -866,7 +923,8 @@ static unsigned long sha1_mb_flusher(struct mcryptd_alg_cstate *cstate)
if (time_before(cur_time, rctx->tag.expire))
break;
kernel_fpu_begin();
sha_ctx = (struct sha1_hash_ctx *) sha1_ctx_mgr_flush(cstate->mgr);
sha_ctx = (struct sha1_hash_ctx *)
sha1_ctx_mgr_flush(cstate->mgr);
kernel_fpu_end();
if (!sha_ctx) {
pr_err("sha1_mb error: nothing got flushed for non-empty list\n");
......@@ -927,7 +985,7 @@ static int __init sha1_mb_mod_init(void)
}
sha1_mb_alg_state.flusher = &sha1_mb_flusher;
err = crypto_register_shash(&sha1_mb_shash_alg);
err = crypto_register_ahash(&sha1_mb_areq_alg);
if (err)
goto err2;
err = crypto_register_ahash(&sha1_mb_async_alg);
......@@ -937,7 +995,7 @@ static int __init sha1_mb_mod_init(void)
return 0;
err1:
crypto_unregister_shash(&sha1_mb_shash_alg);
crypto_unregister_ahash(&sha1_mb_areq_alg);
err2:
for_each_possible_cpu(cpu) {
cpu_state = per_cpu_ptr(sha1_mb_alg_state.alg_cstate, cpu);
......@@ -953,7 +1011,7 @@ static void __exit sha1_mb_mod_fini(void)
struct mcryptd_alg_cstate *cpu_state;
crypto_unregister_ahash(&sha1_mb_async_alg);
crypto_unregister_shash(&sha1_mb_shash_alg);
crypto_unregister_ahash(&sha1_mb_areq_alg);
for_each_possible_cpu(cpu) {
cpu_state = per_cpu_ptr(sha1_mb_alg_state.alg_cstate, cpu);
kfree(cpu_state->mgr);
......
arch/x86/crypto/sha-mb/sha_mb_ctx.h → arch/x86/crypto/sha1-mb/sha1_mb_ctx.h
浏览文件 @ bbce2ad2
......@@ -54,7 +54,7 @@
#ifndef _SHA_MB_CTX_INTERNAL_H
#define _SHA_MB_CTX_INTERNAL_H
#include "sha_mb_mgr.h"
#include "sha1_mb_mgr.h"
#define HASH_UPDATE 0x00
#define HASH_FIRST 0x01
......
arch/x86/crypto/sha-mb/sha1_mb_mgr_init_avx2.c → arch/x86/crypto/sha1-mb/sha1_mb_mgr_init_avx2.c
浏览文件 @ bbce2ad2
......@@ -51,7 +51,7 @@
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include "sha_mb_mgr.h"
#include "sha1_mb_mgr.h"
void sha1_mb_mgr_init_avx2(struct sha1_mb_mgr *state)
{
......
arch/x86/crypto/sha1_ssse3_glue.c
浏览文件 @ bbce2ad2
......@@ -374,3 +374,9 @@ MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("SHA1 Secure Hash Algorithm, Supplemental SSE3 accelerated");
MODULE_ALIAS_CRYPTO("sha1");
MODULE_ALIAS_CRYPTO("sha1-ssse3");
MODULE_ALIAS_CRYPTO("sha1-avx");
MODULE_ALIAS_CRYPTO("sha1-avx2");
#ifdef CONFIG_AS_SHA1_NI
MODULE_ALIAS_CRYPTO("sha1-ni");
#endif
arch/x86/crypto/sha256-mb/Makefile 0 → 100644
浏览文件 @ bbce2ad2
#
# Arch-specific CryptoAPI modules.
#
avx2_supported := $(call as-instr,vpgatherdd %ymm0$(comma)(%eax$(comma)%ymm1\
$(comma)4)$(comma)%ymm2,yes,no)
ifeq ($(avx2_supported),yes)
obj-$(CONFIG_CRYPTO_SHA256_MB) += sha256-mb.o
sha256-mb-y := sha256_mb.o sha256_mb_mgr_flush_avx2.o \
sha256_mb_mgr_init_avx2.o sha256_mb_mgr_submit_avx2.o sha256_x8_avx2.o
endif
arch/x86/crypto/sha256-mb/sha256_mb.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha256-mb/sha256_mb_ctx.h 0 → 100644
浏览文件 @ bbce2ad2
/*
* Header file for multi buffer SHA256 context
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* Contact Information:
* Megha Dey <megha.dey@linux.intel.com>
*
* BSD LICENSE
*
* Copyright(c) 2016 Intel Corporation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef _SHA_MB_CTX_INTERNAL_H
#define _SHA_MB_CTX_INTERNAL_H
#include "sha256_mb_mgr.h"
#define HASH_UPDATE 0x00
#define HASH_FIRST 0x01
#define HASH_LAST 0x02
#define HASH_ENTIRE 0x03
#define HASH_DONE 0x04
#define HASH_FINAL 0x08
#define HASH_CTX_STS_IDLE 0x00
#define HASH_CTX_STS_PROCESSING 0x01
#define HASH_CTX_STS_LAST 0x02
#define HASH_CTX_STS_COMPLETE 0x04
enum hash_ctx_error {
HASH_CTX_ERROR_NONE = 0,
HASH_CTX_ERROR_INVALID_FLAGS = -1,
HASH_CTX_ERROR_ALREADY_PROCESSING = -2,
HASH_CTX_ERROR_ALREADY_COMPLETED = -3,
#ifdef HASH_CTX_DEBUG
HASH_CTX_ERROR_DEBUG_DIGEST_MISMATCH = -4,
#endif
};
#define hash_ctx_user_data(ctx) ((ctx)->user_data)
#define hash_ctx_digest(ctx) ((ctx)->job.result_digest)
#define hash_ctx_processing(ctx) ((ctx)->status & HASH_CTX_STS_PROCESSING)
#define hash_ctx_complete(ctx) ((ctx)->status == HASH_CTX_STS_COMPLETE)
#define hash_ctx_status(ctx) ((ctx)->status)
#define hash_ctx_error(ctx) ((ctx)->error)
#define hash_ctx_init(ctx) \
do { \
(ctx)->error = HASH_CTX_ERROR_NONE; \
(ctx)->status = HASH_CTX_STS_COMPLETE; \
} while (0)
/* Hash Constants and Typedefs */
#define SHA256_DIGEST_LENGTH 8
#define SHA256_LOG2_BLOCK_SIZE 6
#define SHA256_PADLENGTHFIELD_SIZE 8
#ifdef SHA_MB_DEBUG
#define assert(expr) \
do { \
if (unlikely(!(expr))) { \
printk(KERN_ERR "Assertion failed! %s,%s,%s,line=%d\n", \
#expr, __FILE__, __func__, __LINE__); \
} \
} while (0)
#else
#define assert(expr) do {} while (0)
#endif
struct sha256_ctx_mgr {
struct sha256_mb_mgr mgr;
};
/* typedef struct sha256_ctx_mgr sha256_ctx_mgr; */
struct sha256_hash_ctx {
/* Must be at struct offset 0 */
struct job_sha256 job;
/* status flag */
int status;
/* error flag */
int error;
uint32_t total_length;
const void *incoming_buffer;
uint32_t incoming_buffer_length;
uint8_t partial_block_buffer[SHA256_BLOCK_SIZE * 2];
uint32_t partial_block_buffer_length;
void *user_data;
};
#endif
arch/x86/crypto/sha256-mb/sha256_mb_mgr.h 0 → 100644
浏览文件 @ bbce2ad2
/*
* Header file for multi buffer SHA256 algorithm manager
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* Contact Information:
* Megha Dey <megha.dey@linux.intel.com>
*
* BSD LICENSE
*
* Copyright(c) 2016 Intel Corporation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef __SHA_MB_MGR_H
#define __SHA_MB_MGR_H
#include <linux/types.h>
#define NUM_SHA256_DIGEST_WORDS 8
enum job_sts { STS_UNKNOWN = 0,
STS_BEING_PROCESSED = 1,
STS_COMPLETED = 2,
STS_INTERNAL_ERROR = 3,
STS_ERROR = 4
};
struct job_sha256 {
u8 *buffer;
u32 len;
u32 result_digest[NUM_SHA256_DIGEST_WORDS] __aligned(32);
enum job_sts status;
void *user_data;
};
/* SHA256 out-of-order scheduler */
/* typedef uint32_t sha8_digest_array[8][8]; */
struct sha256_args_x8 {
uint32_t digest[8][8];
uint8_t *data_ptr[8];
};
struct sha256_lane_data {
struct job_sha256 *job_in_lane;
};
struct sha256_mb_mgr {
struct sha256_args_x8 args;
uint32_t lens[8];
/* each byte is index (0...7) of unused lanes */
uint64_t unused_lanes;
/* byte 4 is set to FF as a flag */
struct sha256_lane_data ldata[8];
};
#define SHA256_MB_MGR_NUM_LANES_AVX2 8
void sha256_mb_mgr_init_avx2(struct sha256_mb_mgr *state);
struct job_sha256 *sha256_mb_mgr_submit_avx2(struct sha256_mb_mgr *state,
struct job_sha256 *job);
struct job_sha256 *sha256_mb_mgr_flush_avx2(struct sha256_mb_mgr *state);
struct job_sha256 *sha256_mb_mgr_get_comp_job_avx2(struct sha256_mb_mgr *state);
#endif
arch/x86/crypto/sha256-mb/sha256_mb_mgr_datastruct.S 0 → 100644
浏览文件 @ bbce2ad2
/*
* Header file for multi buffer SHA256 algorithm data structure
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* Contact Information:
* Megha Dey <megha.dey@linux.intel.com>
*
* BSD LICENSE
*
* Copyright(c) 2016 Intel Corporation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
# Macros for defining data structures
# Usage example
#START_FIELDS # JOB_AES
### name size align
#FIELD _plaintext, 8, 8 # pointer to plaintext
#FIELD _ciphertext, 8, 8 # pointer to ciphertext
#FIELD _IV, 16, 8 # IV
#FIELD _keys, 8, 8 # pointer to keys
#FIELD _len, 4, 4 # length in bytes
#FIELD _status, 4, 4 # status enumeration
#FIELD _user_data, 8, 8 # pointer to user data
#UNION _union, size1, align1, \
# size2, align2, \
# size3, align3, \
# ...
#END_FIELDS
#%assign _JOB_AES_size _FIELD_OFFSET
#%assign _JOB_AES_align _STRUCT_ALIGN
#########################################################################
# Alternate "struc-like" syntax:
# STRUCT job_aes2
# RES_Q .plaintext, 1
# RES_Q .ciphertext, 1
# RES_DQ .IV, 1
# RES_B .nested, _JOB_AES_SIZE, _JOB_AES_ALIGN
# RES_U .union, size1, align1, \
# size2, align2, \
# ...
# ENDSTRUCT
# # Following only needed if nesting
# %assign job_aes2_size _FIELD_OFFSET
# %assign job_aes2_align _STRUCT_ALIGN
#
# RES_* macros take a name, a count and an optional alignment.
# The count in in terms of the base size of the macro, and the
# default alignment is the base size.
# The macros are:
# Macro Base size
# RES_B 1
# RES_W 2
# RES_D 4
# RES_Q 8
# RES_DQ 16
# RES_Y 32
# RES_Z 64
#
# RES_U defines a union. It's arguments are a name and two or more
# pairs of "size, alignment"
#
# The two assigns are only needed if this structure is being nested
# within another. Even if the assigns are not done, one can still use
# STRUCT_NAME_size as the size of the structure.
#
# Note that for nesting, you still need to assign to STRUCT_NAME_size.
#
# The differences between this and using "struc" directly are that each
# type is implicitly aligned to its natural length (although this can be
# over-ridden with an explicit third parameter), and that the structure
# is padded at the end to its overall alignment.
#
#########################################################################
#ifndef _DATASTRUCT_ASM_
#define _DATASTRUCT_ASM_
#define SZ8 8*SHA256_DIGEST_WORD_SIZE
#define ROUNDS 64*SZ8
#define PTR_SZ 8
#define SHA256_DIGEST_WORD_SIZE 4
#define MAX_SHA256_LANES 8
#define SHA256_DIGEST_WORDS 8
#define SHA256_DIGEST_ROW_SIZE (MAX_SHA256_LANES * SHA256_DIGEST_WORD_SIZE)
#define SHA256_DIGEST_SIZE (SHA256_DIGEST_ROW_SIZE * SHA256_DIGEST_WORDS)
#define SHA256_BLK_SZ 64
# START_FIELDS
.macro START_FIELDS
_FIELD_OFFSET = 0
_STRUCT_ALIGN = 0
.endm
# FIELD name size align
.macro FIELD name size align
_FIELD_OFFSET = (_FIELD_OFFSET + (\align) - 1) & (~ ((\align)-1))
\name = _FIELD_OFFSET
_FIELD_OFFSET = _FIELD_OFFSET + (\size)
.if (\align > _STRUCT_ALIGN)
_STRUCT_ALIGN = \align
.endif
.endm
# END_FIELDS
.macro END_FIELDS
_FIELD_OFFSET = (_FIELD_OFFSET + _STRUCT_ALIGN-1) & (~ (_STRUCT_ALIGN-1))
.endm
########################################################################
.macro STRUCT p1
START_FIELDS
.struc \p1
.endm
.macro ENDSTRUCT
tmp = _FIELD_OFFSET
END_FIELDS
tmp = (_FIELD_OFFSET - %%tmp)
.if (tmp > 0)
.lcomm tmp
.endif
.endstruc
.endm
## RES_int name size align
.macro RES_int p1 p2 p3
name = \p1
size = \p2
align = .\p3
_FIELD_OFFSET = (_FIELD_OFFSET + (align) - 1) & (~ ((align)-1))
.align align
.lcomm name size
_FIELD_OFFSET = _FIELD_OFFSET + (size)
.if (align > _STRUCT_ALIGN)
_STRUCT_ALIGN = align
.endif
.endm
# macro RES_B name, size [, align]
.macro RES_B _name, _size, _align=1
RES_int _name _size _align
.endm
# macro RES_W name, size [, align]
.macro RES_W _name, _size, _align=2
RES_int _name 2*(_size) _align
.endm
# macro RES_D name, size [, align]
.macro RES_D _name, _size, _align=4
RES_int _name 4*(_size) _align
.endm
# macro RES_Q name, size [, align]
.macro RES_Q _name, _size, _align=8
RES_int _name 8*(_size) _align
.endm
# macro RES_DQ name, size [, align]
.macro RES_DQ _name, _size, _align=16
RES_int _name 16*(_size) _align
.endm
# macro RES_Y name, size [, align]
.macro RES_Y _name, _size, _align=32
RES_int _name 32*(_size) _align
.endm
# macro RES_Z name, size [, align]
.macro RES_Z _name, _size, _align=64
RES_int _name 64*(_size) _align
.endm
#endif
########################################################################
#### Define SHA256 Out Of Order Data Structures
########################################################################
START_FIELDS # LANE_DATA
### name size align
FIELD _job_in_lane, 8, 8 # pointer to job object
END_FIELDS
_LANE_DATA_size = _FIELD_OFFSET
_LANE_DATA_align = _STRUCT_ALIGN
########################################################################
START_FIELDS # SHA256_ARGS_X4
### name size align
FIELD _digest, 4*8*8, 4 # transposed digest
FIELD _data_ptr, 8*8, 8 # array of pointers to data
END_FIELDS
_SHA256_ARGS_X4_size = _FIELD_OFFSET
_SHA256_ARGS_X4_align = _STRUCT_ALIGN
_SHA256_ARGS_X8_size = _FIELD_OFFSET
_SHA256_ARGS_X8_align = _STRUCT_ALIGN
#######################################################################
START_FIELDS # MB_MGR
### name size align
FIELD _args, _SHA256_ARGS_X4_size, _SHA256_ARGS_X4_align
FIELD _lens, 4*8, 8
FIELD _unused_lanes, 8, 8
FIELD _ldata, _LANE_DATA_size*8, _LANE_DATA_align
END_FIELDS
_MB_MGR_size = _FIELD_OFFSET
_MB_MGR_align = _STRUCT_ALIGN
_args_digest = _args + _digest
_args_data_ptr = _args + _data_ptr
#######################################################################
START_FIELDS #STACK_FRAME
### name size align
FIELD _data, 16*SZ8, 1 # transposed digest
FIELD _digest, 8*SZ8, 1 # array of pointers to data
FIELD _ytmp, 4*SZ8, 1
FIELD _rsp, 8, 1
END_FIELDS
_STACK_FRAME_size = _FIELD_OFFSET
_STACK_FRAME_align = _STRUCT_ALIGN
#######################################################################
########################################################################
#### Define constants
########################################################################
#define STS_UNKNOWN 0
#define STS_BEING_PROCESSED 1
#define STS_COMPLETED 2
########################################################################
#### Define JOB_SHA256 structure
########################################################################
START_FIELDS # JOB_SHA256
### name size align
FIELD _buffer, 8, 8 # pointer to buffer
FIELD _len, 8, 8 # length in bytes
FIELD _result_digest, 8*4, 32 # Digest (output)
FIELD _status, 4, 4
FIELD _user_data, 8, 8
END_FIELDS
_JOB_SHA256_size = _FIELD_OFFSET
_JOB_SHA256_align = _STRUCT_ALIGN
arch/x86/crypto/sha256-mb/sha256_mb_mgr_flush_avx2.S 0 → 100644
浏览文件 @ bbce2ad2
/*
* Flush routine for SHA256 multibuffer
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* Contact Information:
* Megha Dey <megha.dey@linux.intel.com>
*
* BSD LICENSE
*
* Copyright(c) 2016 Intel Corporation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <linux/linkage.h>
#include <asm/frame.h>
#include "sha256_mb_mgr_datastruct.S"
.extern sha256_x8_avx2
#LINUX register definitions
#define arg1 %rdi
#define arg2 %rsi
# Common register definitions
#define state arg1
#define job arg2
#define len2 arg2
# idx must be a register not clobberred by sha1_mult
#define idx %r8
#define DWORD_idx %r8d
#define unused_lanes %rbx
#define lane_data %rbx
#define tmp2 %rbx
#define tmp2_w %ebx
#define job_rax %rax
#define tmp1 %rax
#define size_offset %rax
#define tmp %rax
#define start_offset %rax
#define tmp3 %arg1
#define extra_blocks %arg2
#define p %arg2
.macro LABEL prefix n
\prefix\n\():
.endm
.macro JNE_SKIP i
jne skip_\i
.endm
.altmacro
.macro SET_OFFSET _offset
offset = \_offset
.endm
.noaltmacro
# JOB_SHA256* sha256_mb_mgr_flush_avx2(MB_MGR *state)
# arg 1 : rcx : state
ENTRY(sha256_mb_mgr_flush_avx2)
FRAME_BEGIN
push %rbx
# If bit (32+3) is set, then all lanes are empty
mov _unused_lanes(state), unused_lanes
bt $32+3, unused_lanes
jc return_null
# find a lane with a non-null job
xor idx, idx
offset = (_ldata + 1 * _LANE_DATA_size + _job_in_lane)
cmpq $0, offset(state)
cmovne one(%rip), idx
offset = (_ldata + 2 * _LANE_DATA_size + _job_in_lane)
cmpq $0, offset(state)
cmovne two(%rip), idx
offset = (_ldata + 3 * _LANE_DATA_size + _job_in_lane)
cmpq $0, offset(state)
cmovne three(%rip), idx
offset = (_ldata + 4 * _LANE_DATA_size + _job_in_lane)
cmpq $0, offset(state)
cmovne four(%rip), idx
offset = (_ldata + 5 * _LANE_DATA_size + _job_in_lane)
cmpq $0, offset(state)
cmovne five(%rip), idx
offset = (_ldata + 6 * _LANE_DATA_size + _job_in_lane)
cmpq $0, offset(state)
cmovne six(%rip), idx
offset = (_ldata + 7 * _LANE_DATA_size + _job_in_lane)
cmpq $0, offset(state)
cmovne seven(%rip), idx
# copy idx to empty lanes
copy_lane_data:
offset = (_args + _data_ptr)
mov offset(state,idx,8), tmp
I = 0
.rep 8
offset = (_ldata + I * _LANE_DATA_size + _job_in_lane)
cmpq $0, offset(state)
.altmacro
JNE_SKIP %I
offset = (_args + _data_ptr + 8*I)
mov tmp, offset(state)
offset = (_lens + 4*I)
movl $0xFFFFFFFF, offset(state)
LABEL skip_ %I
I = (I+1)
.noaltmacro
.endr
# Find min length
vmovdqa _lens+0*16(state), %xmm0
vmovdqa _lens+1*16(state), %xmm1
vpminud %xmm1, %xmm0, %xmm2 # xmm2 has {D,C,B,A}
vpalignr $8, %xmm2, %xmm3, %xmm3 # xmm3 has {x,x,D,C}
vpminud %xmm3, %xmm2, %xmm2 # xmm2 has {x,x,E,F}
vpalignr $4, %xmm2, %xmm3, %xmm3 # xmm3 has {x,x,x,E}
vpminud %xmm3, %xmm2, %xmm2 # xmm2 has min val in low dword
vmovd %xmm2, DWORD_idx
mov idx, len2
and $0xF, idx
shr $4, len2
jz len_is_0
vpand clear_low_nibble(%rip), %xmm2, %xmm2
vpshufd $0, %xmm2, %xmm2
vpsubd %xmm2, %xmm0, %xmm0
vpsubd %xmm2, %xmm1, %xmm1
vmovdqa %xmm0, _lens+0*16(state)
vmovdqa %xmm1, _lens+1*16(state)
# "state" and "args" are the same address, arg1
# len is arg2
call sha256_x8_avx2
# state and idx are intact
len_is_0:
# process completed job "idx"
imul $_LANE_DATA_size, idx, lane_data
lea _ldata(state, lane_data), lane_data
mov _job_in_lane(lane_data), job_rax
movq $0, _job_in_lane(lane_data)
movl $STS_COMPLETED, _status(job_rax)
mov _unused_lanes(state), unused_lanes
shl $4, unused_lanes
or idx, unused_lanes
mov unused_lanes, _unused_lanes(state)
movl $0xFFFFFFFF, _lens(state,idx,4)
vmovd _args_digest(state , idx, 4) , %xmm0
vpinsrd $1, _args_digest+1*32(state, idx, 4), %xmm0, %xmm0
vpinsrd $2, _args_digest+2*32(state, idx, 4), %xmm0, %xmm0
vpinsrd $3, _args_digest+3*32(state, idx, 4), %xmm0, %xmm0
vmovd _args_digest+4*32(state, idx, 4), %xmm1
vpinsrd $1, _args_digest+5*32(state, idx, 4), %xmm1, %xmm1
vpinsrd $2, _args_digest+6*32(state, idx, 4), %xmm1, %xmm1
vpinsrd $3, _args_digest+7*32(state, idx, 4), %xmm1, %xmm1
vmovdqu %xmm0, _result_digest(job_rax)
offset = (_result_digest + 1*16)
vmovdqu %xmm1, offset(job_rax)
return:
pop %rbx
FRAME_END
ret
return_null:
xor job_rax, job_rax
jmp return
ENDPROC(sha256_mb_mgr_flush_avx2)
##############################################################################
.align 16
ENTRY(sha256_mb_mgr_get_comp_job_avx2)
push %rbx
## if bit 32+3 is set, then all lanes are empty
mov _unused_lanes(state), unused_lanes
bt $(32+3), unused_lanes
jc .return_null
# Find min length
vmovdqa _lens(state), %xmm0
vmovdqa _lens+1*16(state), %xmm1
vpminud %xmm1, %xmm0, %xmm2 # xmm2 has {D,C,B,A}
vpalignr $8, %xmm2, %xmm3, %xmm3 # xmm3 has {x,x,D,C}
vpminud %xmm3, %xmm2, %xmm2 # xmm2 has {x,x,E,F}
vpalignr $4, %xmm2, %xmm3, %xmm3 # xmm3 has {x,x,x,E}
vpminud %xmm3, %xmm2, %xmm2 # xmm2 has min val in low dword
vmovd %xmm2, DWORD_idx
test $~0xF, idx
jnz .return_null
# process completed job "idx"
imul $_LANE_DATA_size, idx, lane_data
lea _ldata(state, lane_data), lane_data
mov _job_in_lane(lane_data), job_rax
movq $0, _job_in_lane(lane_data)
movl $STS_COMPLETED, _status(job_rax)
mov _unused_lanes(state), unused_lanes
shl $4, unused_lanes
or idx, unused_lanes
mov unused_lanes, _unused_lanes(state)
movl $0xFFFFFFFF, _lens(state, idx, 4)
vmovd _args_digest(state, idx, 4), %xmm0
vpinsrd $1, _args_digest+1*32(state, idx, 4), %xmm0, %xmm0
vpinsrd $2, _args_digest+2*32(state, idx, 4), %xmm0, %xmm0
vpinsrd $3, _args_digest+3*32(state, idx, 4), %xmm0, %xmm0
movl _args_digest+4*32(state, idx, 4), tmp2_w
vpinsrd $1, _args_digest+5*32(state, idx, 4), %xmm1, %xmm1
vpinsrd $2, _args_digest+6*32(state, idx, 4), %xmm1, %xmm1
vpinsrd $3, _args_digest+7*32(state, idx, 4), %xmm1, %xmm1
vmovdqu %xmm0, _result_digest(job_rax)
movl tmp2_w, _result_digest+1*16(job_rax)
pop %rbx
ret
.return_null:
xor job_rax, job_rax
pop %rbx
ret
ENDPROC(sha256_mb_mgr_get_comp_job_avx2)
.data
.align 16
clear_low_nibble:
.octa 0x000000000000000000000000FFFFFFF0
one:
.quad 1
two:
.quad 2
three:
.quad 3
four:
.quad 4
five:
.quad 5
six:
.quad 6
seven:
.quad 7
arch/x86/crypto/sha256-mb/sha256_mb_mgr_init_avx2.c 0 → 100644
浏览文件 @ bbce2ad2
/*
* Initialization code for multi buffer SHA256 algorithm for AVX2
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* Contact Information:
* Megha Dey <megha.dey@linux.intel.com>
*
* BSD LICENSE
*
* Copyright(c) 2016 Intel Corporation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include "sha256_mb_mgr.h"
void sha256_mb_mgr_init_avx2(struct sha256_mb_mgr *state)
{
unsigned int j;
state->unused_lanes = 0xF76543210ULL;
for (j = 0; j < 8; j++) {
state->lens[j] = 0xFFFFFFFF;
state->ldata[j].job_in_lane = NULL;
}
}
arch/x86/crypto/sha256-mb/sha256_mb_mgr_submit_avx2.S 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha256-mb/sha256_x8_avx2.S 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha256_ssse3_glue.c
浏览文件 @ bbce2ad2
......@@ -427,4 +427,14 @@ MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("SHA256 Secure Hash Algorithm, Supplemental SSE3 accelerated");
MODULE_ALIAS_CRYPTO("sha256");
MODULE_ALIAS_CRYPTO("sha256-ssse3");
MODULE_ALIAS_CRYPTO("sha256-avx");
MODULE_ALIAS_CRYPTO("sha256-avx2");
MODULE_ALIAS_CRYPTO("sha224");
MODULE_ALIAS_CRYPTO("sha224-ssse3");
MODULE_ALIAS_CRYPTO("sha224-avx");
MODULE_ALIAS_CRYPTO("sha224-avx2");
#ifdef CONFIG_AS_SHA256_NI
MODULE_ALIAS_CRYPTO("sha256-ni");
MODULE_ALIAS_CRYPTO("sha224-ni");
#endif
arch/x86/crypto/sha512-mb/Makefile 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha512-mb/sha512_mb.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha512-mb/sha512_mb_ctx.h 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha512-mb/sha512_mb_mgr.h 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha512-mb/sha512_mb_mgr_datastruct.S 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha512-mb/sha512_mb_mgr_flush_avx2.S 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha512-mb/sha512_mb_mgr_init_avx2.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha512-mb/sha512_mb_mgr_submit_avx2.S 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha512-mb/sha512_x4_avx2.S 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
arch/x86/crypto/sha512_ssse3_glue.c
浏览文件 @ bbce2ad2
......@@ -346,4 +346,10 @@ MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("SHA512 Secure Hash Algorithm, Supplemental SSE3 accelerated");
MODULE_ALIAS_CRYPTO("sha512");
MODULE_ALIAS_CRYPTO("sha512-ssse3");
MODULE_ALIAS_CRYPTO("sha512-avx");
MODULE_ALIAS_CRYPTO("sha512-avx2");
MODULE_ALIAS_CRYPTO("sha384");
MODULE_ALIAS_CRYPTO("sha384-ssse3");
MODULE_ALIAS_CRYPTO("sha384-avx");
MODULE_ALIAS_CRYPTO("sha384-avx2");
crypto/Kconfig
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/Makefile
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/ablk_helper.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/ablkcipher.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/aead.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/ahash.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/algapi.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/authenc.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/authencesn.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/blkcipher.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/ccm.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/chacha20poly1305.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/chainiv.c 已删除 100644 → 0
浏览文件 @ 0f776dc3
此差异已折叠。
crypto/cryptd.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/crypto_null.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/crypto_user.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/ctr.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/cts.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/dh.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/dh_helper.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/drbg.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/ecc.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/ecc.h 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/ecc_curve_defs.h 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/ecdh.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/ecdh_helper.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/echainiv.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/eseqiv.c 已删除 100644 → 0
浏览文件 @ 0f776dc3
此差异已折叠。
crypto/gcm.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/jitterentropy-kcapi.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/kpp.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/mcryptd.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/rsa-pkcs1pad.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/rsa.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/rsa_helper.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/rsaprivkey.asn1
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/scatterwalk.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/seqiv.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/sha3_generic.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/skcipher.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/tcrypt.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/testmgr.c
浏览文件 @ bbce2ad2
此差异已折叠。
crypto/testmgr.h
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/char/hw_random/Kconfig
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/char/hw_random/Makefile
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/char/hw_random/exynos-rng.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/char/hw_random/meson-rng.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/char/hw_random/omap-rng.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/char/hw_random/stm32-rng.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/bfin_crc.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/Kconfig
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/Makefile
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/caamhash.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/caampkc.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/caampkc.h 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/compat.h
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/ctrl.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/desc.h
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/desc_constr.h
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/jr.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/pdb.h
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/pkc_desc.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/regs.h
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/caam/sg_sw_sec4.h
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/ccp/ccp-crypto.h
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/marvell/cesa.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/marvell/cesa.h
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/marvell/cipher.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/marvell/hash.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/marvell/tdma.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/mxs-dcp.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/nx/nx.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/omap-aes.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/omap-des.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/omap-sham.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/picoxcell_crypto.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/qat/Kconfig
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/qce/ablkcipher.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/qce/cipher.h
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/s5p-sss.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/sahara.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/talitos.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/ux500/cryp/Makefile
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/ux500/hash/Makefile
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/vmx/.gitignore 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/vmx/Makefile
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/vmx/aes_xts.c 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/vmx/aesp8-ppc.h
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/vmx/aesp8-ppc.pl
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/crypto/vmx/vmx.c
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/usb/misc/Kconfig
浏览文件 @ bbce2ad2
此差异已折叠。
drivers/usb/misc/chaoskey.c
浏览文件 @ bbce2ad2
此差异已折叠。
include/asm-generic/io.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/asm-generic/iomap.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/aead.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/algapi.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/cryptd.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/dh.h 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/drbg.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/ecdh.h 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/internal/aead.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/internal/geniv.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/internal/hash.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/internal/kpp.h 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/internal/rsa.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/internal/skcipher.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/kpp.h 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/mcryptd.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/null.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/scatterwalk.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/sha3.h 0 → 100644
浏览文件 @ bbce2ad2
此差异已折叠。
include/crypto/skcipher.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/linux/crypto.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/linux/mpi.h
浏览文件 @ bbce2ad2
此差异已折叠。
include/uapi/linux/cryptouser.h
浏览文件 @ bbce2ad2
此差异已折叠。
lib/digsig.c
浏览文件 @ bbce2ad2
此差异已折叠。
lib/mpi/mpicoder.c
浏览文件 @ bbce2ad2
此差异已折叠。
security/keys/big_key.c
浏览文件 @ bbce2ad2
此差异已折叠。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或