Blkcipher touching FPU need to be enclosed by kernel_fpu_begin() and
kernel_fpu_end(). If they are invoked in cipher algorithm
implementation, they will be invoked for each block, so that
performance will be hurt, because they are "slow" operations. This
patch implements "fpu" template, which makes these operations to be
invoked for each request.
Signed-off-by: NHuang Ying <ying.huang@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Signed-off-by: NGeert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Signed-off-by: NGeert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
Cc: James Morris <jmorris@namei.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The current "comp" crypto interface supports one-shot (de)compression only,
i.e. the whole data buffer to be (de)compressed must be passed at once, and
the whole (de)compressed data buffer will be received at once.
In several use-cases (e.g. compressed file systems that store files in big
compressed blocks), this workflow is not suitable.
Furthermore, the "comp" type doesn't provide for the configuration of
(de)compression parameters, and always allocates workspace memory for both
compression and decompression, which may waste memory.

To solve this, add a "pcomp" partial (de)compression interface that provides
the following operations:
  - crypto_compress_{init,update,final}() for compression,
  - crypto_decompress_{init,update,final}() for decompression,
  - crypto_{,de}compress_setup(), to configure (de)compression parameters
    (incl. allocating workspace memory).

The (de)compression methods take a struct comp_request, which was mimicked
after the z_stream object in zlib, and contains buffer pointer and length
pairs for input and output.

The setup methods take an opaque parameter pointer and length pair. Parameters
are supposed to be encoded using netlink attributes, whose meanings depend on
the actual (name of the) (de)compression algorithm.
Signed-off-by: NGeert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

keventd_wq has potential starvation problem, so use dedicated
kcrypto_wq instead.
Signed-off-by: NHuang Ying <ying.huang@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Original cryptd thread implementation has scalability issue, this
patch solve the issue with a per-CPU thread implementation.

struct cryptd_queue is defined to be a per-CPU queue, which holds one
struct cryptd_cpu_queue for each CPU. In struct cryptd_cpu_queue, a
struct crypto_queue holds all requests for the CPU, a struct
work_struct is used to run all requests for the CPU.

Testing based on dm-crypt on an Intel Core 2 E6400 (two cores) machine
shows 19.2% performance gain. The testing script is as follow:

-------------------- script begin ---------------------------
#!/bin/sh

dmc_create()
{
        # Create a crypt device using dmsetup
        dmsetup create $2 --table "0 `blockdev --getsize $1` crypt cbc(aes-asm)?cryptd?plain:plain babebabebabebabebabebabebabebabe 0 $1 0"
}

dmsetup remove crypt0
dmsetup remove crypt1

dd if=/dev/zero of=/dev/ram0 bs=1M count=4 >& /dev/null
dd if=/dev/zero of=/dev/ram1 bs=1M count=4 >& /dev/null

dmc_create /dev/ram0 crypt0
dmc_create /dev/ram1 crypt1

cat >tr.sh <<EOF
#!/bin/sh

for n in \$(seq 10); do
        dd if=/dev/dm-0 of=/dev/null >& /dev/null &
        dd if=/dev/dm-1 of=/dev/null >& /dev/null &
done
wait
EOF

for n in $(seq 10); do
        /usr/bin/time sh tr.sh
done
rm tr.sh
-------------------- script end   ---------------------------

The separator of dm-crypt parameter is changed from "-" to "?", because
"-" is used in some cipher driver name too, and cryptds need to specify
cipher driver name instead of cipher name.

The test result on an Intel Core2 E6400 (two cores) is as follow:

without patch:
-----------------wo begin --------------------------
0.04user 0.38system 0:00.39elapsed 107%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6566minor)pagefaults 0swaps
0.07user 0.35system 0:00.35elapsed 121%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6567minor)pagefaults 0swaps
0.06user 0.34system 0:00.30elapsed 135%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6562minor)pagefaults 0swaps
0.05user 0.37system 0:00.36elapsed 119%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6607minor)pagefaults 0swaps
0.06user 0.36system 0:00.35elapsed 120%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6562minor)pagefaults 0swaps
0.05user 0.37system 0:00.31elapsed 136%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6594minor)pagefaults 0swaps
0.04user 0.34system 0:00.30elapsed 126%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6597minor)pagefaults 0swaps
0.06user 0.32system 0:00.31elapsed 125%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6571minor)pagefaults 0swaps
0.06user 0.34system 0:00.31elapsed 134%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6581minor)pagefaults 0swaps
0.05user 0.38system 0:00.31elapsed 138%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6600minor)pagefaults 0swaps
-----------------wo end   --------------------------


with patch:
------------------w begin --------------------------
0.02user 0.31system 0:00.24elapsed 141%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6554minor)pagefaults 0swaps
0.05user 0.34system 0:00.31elapsed 127%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6606minor)pagefaults 0swaps
0.07user 0.33system 0:00.26elapsed 155%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6559minor)pagefaults 0swaps
0.07user 0.32system 0:00.26elapsed 151%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6562minor)pagefaults 0swaps
0.05user 0.34system 0:00.26elapsed 150%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6603minor)pagefaults 0swaps
0.03user 0.36system 0:00.31elapsed 124%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6562minor)pagefaults 0swaps
0.04user 0.35system 0:00.26elapsed 147%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6586minor)pagefaults 0swaps
0.03user 0.37system 0:00.27elapsed 146%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6562minor)pagefaults 0swaps
0.04user 0.36system 0:00.26elapsed 154%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6594minor)pagefaults 0swaps
0.04user 0.35system 0:00.26elapsed 154%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6557minor)pagefaults 0swaps
------------------w end   --------------------------

The middle value of elapsed time is:
wo cryptwq: 0.31
w  cryptwq: 0.26

The performance gain is about (0.31-0.26)/0.26 = 0.192.
Signed-off-by: NHuang Ying <ying.huang@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use dedicated workqueue for crypto subsystem

A dedicated workqueue named kcrypto_wq is created to be used by crypto
subsystem. The system shared keventd_wq is not suitable for
encryption/decryption, because of potential starvation problem.
Signed-off-by: NHuang Ying <ying.huang@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Intel AES-NI is a new set of Single Instruction Multiple Data (SIMD)
instructions that are going to be introduced in the next generation of
Intel processor, as of 2009. These instructions enable fast and secure
data encryption and decryption, using the Advanced Encryption Standard
(AES), defined by FIPS Publication number 197.  The architecture
introduces six instructions that offer full hardware support for
AES. Four of them support high performance data encryption and
decryption, and the other two instructions support the AES key
expansion procedure.

The white paper can be downloaded from:

http://softwarecommunity.intel.com/isn/downloads/intelavx/AES-Instructions-Set_WP.pdf

AES may be used in soft_irq context, but MMX/SSE context can not be
touched safely in soft_irq context. So in_interrupt() is checked, if
in IRQ or soft_irq context, the general x86_64 implementation are used
instead.
Signed-off-by: NHuang Ying <ying.huang@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes sha512 and sha384 to the new shash interface.
Signed-off-by: NAdrian-Ken Rueegsegger <ken@codelabs.ch>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes michael_mic to the new shash interface.
Signed-off-by: NAdrian-Ken Rueegsegger <ken@codelabs.ch>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes wp512, wp384 and wp256 to the new shash interface.
Signed-off-by: NAdrian-Ken Rueegsegger <ken@codelabs.ch>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes tgr192, tgr160 and tgr128 to the new shash interface.
Signed-off-by: NAdrian-Ken Rueegsegger <ken@codelabs.ch>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes sha256 and sha224 to the new shash interface.
Signed-off-by: NAdrian-Ken Rueegsegger <ken@codelabs.ch>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes md5 to the new shash interface.
Signed-off-by: NAdrian-Ken Rueegsegger <ken@codelabs.ch>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes md4 to the new shash interface.
Signed-off-by: NAdrian-Ken Rueegsegger <ken@codelabs.ch>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes sha1 to the new shash interface.
Signed-off-by: NAdrian-Ken Rueegsegger <ken@codelabs.ch>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes rmd320 to the new shash interface.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes rmd256 to the new shash interface.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes rmd160 to the new shash interface.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes rmd128 to the new shash interface.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch changes digest_null to the new shash interface.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch swaps the role of libcrc32c and crc32c.  Previously
the implementation was in libcrc32c and crc32c was a wrapper.
Now the code is in crc32c and libcrc32c just calls the crypto
layer.

The reason for the change is to tap into the algorithm selection
capability of the crypto API so that optimised implementations
such as the one utilising Intel's CRC32C instruction can be
used where available.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If we have at least one algorithm built-in then it no longer makes
sense to have the testing framework, and hence cryptomgr to be a
module.  It should be either on or off, i.e., built-in or disabled.

This just happens to stop a potential runaway modprobe loop that
seems to trigger on at least one distro.

With fixes from Evgeniy Polyakov.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch makes the IV generators use the new RNG interface so
that the user can pick an RNG other than the default get_random_bytes.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch adds a random number generator interface as well as a
cryptographic pseudo-random number generator based on AES.  It is
meant to be used in cases where a deterministic CPRNG is required.

One of the first applications will be as an input in the IPsec IV
generation process.
Signed-off-by: NNeil Horman <nhorman@tuxdriver.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add the ability to turn FIPS-compliant mode on or off at boot

In order to be FIPS compliant, several check may need to be preformed that may
be construed as unusefull in a non-compliant mode.  This patch allows us to set
a kernel flag incating that we are running in a fips-compliant mode from boot
up.  It also exports that mode information to user space via a sysctl
(/proc/sys/crypto/fips_enabled).

Tested successfully by me.
Signed-off-by: NNeil Horman <nhorman@tuxdriver.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch moves the newly created alg_test infrastructure into
cryptomgr.  This shall allow us to use it for testing at algorithm
registrations.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

From NHM processor onward, Intel processors can support hardware accelerated
CRC32c algorithm with the new CRC32 instruction in SSE 4.2 instruction set.
The patch detects the availability of the feature, and chooses the most proper
way to calculate CRC32c checksum.
Byte code instructions are used for compiler compatibility.
No MMX / XMM registers is involved in the implementation.
Signed-off-by: NAustin Zhang <austin.zhang@intel.com>
Signed-off-by: NKent Liu <kent.liu@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Instead of tabs there were two spaces.
Signed-off-by: NAdrian Bunk <bunk@kernel.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch is clearly not ready yet for prime time.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch reimplements crc32c using the ahash interface.  This
allows one tfm to be used by an unlimited number of users provided
that they all use the same key (which all current crc32c users do).
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch adds a cryptographic pseudo-random number generator
based on CTR(AES-128).  It is meant to be used in cases where a
deterministic CPRNG is required.

One of the first applications will be as an input in the IPsec IV
generation process.
Signed-off-by: NNeil Horman <nhorman@tuxdriver.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch adds asynchronous hash support to crypto daemon.
Signed-off-by: NLoc Ho <lho@amcc.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch adds Kconfig entries for RIPEMD-256 and RIPEMD-320.
Signed-off-by: NAdrian-Ken Rueegsegger <rueegsegger@swiss-it.ch>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch adds Kconfig entries for RIPEMD-128 and RIPEMD-160.
Signed-off-by: NAdrian-Ken Rueegsegger <rueegsegger@swiss-it.ch>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Ciphers, block modes, name it, are grouped together and sorted.
Signed-off-by: NSebastian Siewior <sebastian@breakpoint.cc>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Signed-off-by: NSebastian Siewior <sebastian@breakpoint.cc>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Implement CTS wrapper for CBC mode required for support of AES
encryption support for Kerberos (rfc3962).
Signed-off-by: NKevin Coffman <kwc@citi.umich.edu>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The authenc algorithm requires BLKCIPHER to be present.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch fixes the following build error caused by commit 
3631c650:

<--  snip  -->

...
  LD      .tmp_vmlinux1
crypto/built-in.o: In function `skcipher_null_crypt':
crypto_null.c:(.text+0x3d14): undefined reference to `blkcipher_walk_virt'
crypto_null.c:(.text+0x3d14): relocation truncated to fit: R_MIPS_26 against `blkcipher_walk_virt'
crypto/built-in.o: In function `$L32':
crypto_null.c:(.text+0x3d54): undefined reference to `blkcipher_walk_done'
crypto_null.c:(.text+0x3d54): relocation truncated to fit: R_MIPS_26 against `blkcipher_walk_done'
crypto/built-in.o:(.data+0x2e8): undefined reference to `crypto_blkcipher_type'
make[1]: *** [.tmp_vmlinux1] Error 1

<--  snip  -->
Signed-off-by: NAdrian Bunk <bunk@kernel.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>