Original implementation of aesni_cbc_dec do not save IV if input
length % 4 == 0. This will make decryption of next block failed.
Signed-off-by: NHuang Ying <ying.huang@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Because kernel_fpu_begin() and kernel_fpu_end() operations are too
slow, the performance gain of general mode implementation + aes-aesni
is almost all compensated.

The AES-NI support for more modes are implemented as follow:

- Add a new AES algorithm implementation named __aes-aesni without
  kernel_fpu_begin/end()

- Use fpu(<mode>(AES)) to provide kenrel_fpu_begin/end() invoking

- Add <mode>(AES) ablkcipher, which uses cryptd(fpu(<mode>(AES))) to
  defer cryption to cryptd context in soft_irq context.

Now the ctr, lrw, pcbc and xts support are added.

Performance testing based on dm-crypt shows that cryption time can be
reduced to 50% of general mode implementation + aes-aesni implementation.
Signed-off-by: NHuang Ying <ying.huang@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Blkcipher touching FPU need to be enclosed by kernel_fpu_begin() and
kernel_fpu_end(). If they are invoked in cipher algorithm
implementation, they will be invoked for each block, so that
performance will be hurt, because they are "slow" operations. This
patch implements "fpu" template, which makes these operations to be
invoked for each request.
Signed-off-by: NHuang Ying <ying.huang@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Intel AES-NI is a new set of Single Instruction Multiple Data (SIMD)
instructions that are going to be introduced in the next generation of
Intel processor, as of 2009. These instructions enable fast and secure
data encryption and decryption, using the Advanced Encryption Standard
(AES), defined by FIPS Publication number 197.  The architecture
introduces six instructions that offer full hardware support for
AES. Four of them support high performance data encryption and
decryption, and the other two instructions support the AES key
expansion procedure.

The white paper can be downloaded from:

http://softwarecommunity.intel.com/isn/downloads/intelavx/AES-Instructions-Set_WP.pdf

AES may be used in soft_irq context, but MMX/SSE context can not be
touched safely in soft_irq context. So in_interrupt() is checked, if
in IRQ or soft_irq context, the general x86_64 implementation are used
instead.
Signed-off-by: NHuang Ying <ying.huang@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Intel AES-NI AES acceleration instructions touch XMM state, to use
that in soft_irq context, general x86 AES implementation is used as
fallback. The first parameter is changed from struct crypto_tfm * to
struct crypto_aes_ctx * to make it easier to deal with 16 bytes
alignment requirement of AES-NI implementation.
Signed-off-by: NHuang Ying <ying.huang@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The Intel AES-NI AES acceleration instructions need key_enc, key_dec
in struct crypto_aes_ctx to be 16 byte aligned, it make this easier to
move key_length to be the last one.
Signed-off-by: NHuang Ying <ying.huang@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes crc32c-intel to the new shash interface.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The original copyright head for crc32c-intel.c is incorrect. Please merge
the patch to update it.
Signed-Off-By: NKent Liu <kent.liu@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

From NHM processor onward, Intel processors can support hardware accelerated
CRC32c algorithm with the new CRC32 instruction in SSE 4.2 instruction set.
The patch detects the availability of the feature, and chooses the most proper
way to calculate CRC32c checksum.
Byte code instructions are used for compiler compatibility.
No MMX / XMM registers is involved in the implementation.
Signed-off-by: NAustin Zhang <austin.zhang@intel.com>
Signed-off-by: NKent Liu <kent.liu@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The return parameter isn't used remove it.
Signed-off-by: NSebastian Siewior <sebastian@breakpoint.cc>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

There is almost no difference between 32 & 64 bit glue code.
Signed-off-by: NSebastian Siewior <sebastian@breakpoint.cc>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This is the x86-64 version of the Salsa20 stream cipher algorithm. The
original assembly code came from
<http://cr.yp.to/snuffle/salsa20/amd64-3/salsa20.s>. It has been
reformatted for clarity.
Signed-off-by: NTan Swee Heng <thesweeheng@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch contains the salsa20-i586 implementation. The original
assembly code came from
<http://cr.yp.to/snuffle/salsa20/x86-pm/salsa20.s>. I have reformatted
it (added indents) so that it matches the other algorithms in
arch/x86/crypto.
Signed-off-by: NTan Swee Heng <thesweeheng@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

32 bit and 64 bit glue code is using (now) the same
piece code. This patch unifies them.
Signed-off-by: NSebastian Siewior <sebastian@breakpoint.cc>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The setkey() function can be shared with the generic algorithm.
Signed-off-by: NSebastian Siewior <sebastian@breakpoint.cc>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The setkey() function can be shared with the generic algorithm.
Signed-off-by: NSebastian Siewior <sebastian@breakpoint.cc>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This three defines are used in all AES related hardware.
Signed-off-by: NSebastian Siewior <sebastian@breakpoint.cc>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Signed-off-by: NIngo Molnar <mingo@elte.hu>

Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Signed-off-by: NIngo Molnar <mingo@elte.hu>

Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Signed-off-by: NIngo Molnar <mingo@elte.hu>