Restructure squashfs_readpage() splitting it into separate
functions for datablocks, fragments and sparse blocks.

Move the memcpying (from squashfs cache entry) implementation of
squashfs_readpage_block into file_cache.c

This allows different implementations to be supported.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>
Reviewed-by: NMinchan Kim <minchan@kernel.org>

Further generalise the decompressors by adding a page handler
abstraction.  This adds helpers to allow the decompressors
to access and process the output buffers in an implementation
independant manner.

This allows different types of output buffer to be passed
to the decompressors, with the implementation specific
aspects handled at decompression time, but without the
knowledge being held in the decompressor wrapper code.

This will allow the decompressors to handle Squashfs
cache buffers, and page cache pages.

This patch adds the abstraction and an implementation for
the caches.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>
Reviewed-by: NMinchan Kim <minchan@kernel.org>

Add a multi-threaded decompression implementation which uses
percpu variables.

Using percpu variables has advantages and disadvantages over
implementations which do not use percpu variables.

Advantages:
  * the nature of percpu variables ensures decompression is
    load-balanced across the multiple cores.
  * simplicity.

Disadvantages: it limits decompression to one thread per core.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

Now squashfs have used for only one stream buffer for decompression
so it hurts parallel read performance so this patch supports
multiple decompressor to enhance performance parallel I/O.

Four 1G file dd read on KVM machine which has 2 CPU and 4G memory.

dd if=test/test1.dat of=/dev/null &
dd if=test/test2.dat of=/dev/null &
dd if=test/test3.dat of=/dev/null &
dd if=test/test4.dat of=/dev/null &

old : 1m39s -> new : 9s

* From v1
  * Change comp_strm with decomp_strm - Phillip
  * Change/add comments - Phillip
Signed-off-by: NMinchan Kim <minchan@kernel.org>
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

The decompressor interface and code was written from
the point of view of single-threaded operation.  In doing
so it mixed a lot of single-threaded implementation specific
aspects into the decompressor code and elsewhere which makes it
difficult to seamlessly support multiple different decompressor
implementations.

This patch does the following:

1.  It removes compressor_options parsing from the decompressor
    init() function.  This allows the decompressor init() function
    to be dynamically called to instantiate multiple decompressors,
    without the compressor options needing to be read and parsed each
    time.

2.  It moves threading and all sleeping operations out of the
    decompressors.  In doing so, it makes the decompressors
    non-blocking wrappers which only deal with interfacing with
    the decompressor implementation.

3. It splits decompressor.[ch] into decompressor generic functions
   in decompressor.[ch], and moves the single threaded
   decompressor implementation into decompressor_single.c.

The result of this patch is Squashfs should now be able to
support multiple decompressors by adding new decompressor_xxx.c
files with specialised implementations of the functions in
decompressor_single.c
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>
Reviewed-by: NMinchan Kim <minchan@kernel.org>

We read the type field from disk.  This value should be sanity
checked for correctness to avoid an out of bounds access when
reading the squashfs_filetype_table array.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

We read the size (of the name) field from disk.  This value should
be sanity checked for correctness to avoid blindly reading
huge amounts of unnecessary data from disk on corruption.

Note, here we're not actually reading the name into a buffer, but
skipping it, and so corruption doesn't cause buffer overflow, merely
lots of unnecessary amounts of data to be read.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

The dir_count and size fields when read from disk are sanity
checked for correctness.  However, the sanity checks only check the
values are not greater than expected.  As dir_count and size were
incorrectly defined as signed ints, this can lead to corrupted values
appearing as negative which are not trapped.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

The dir_count and size fields when read from disk are sanity
checked for correctness.  However, the sanity checks only check the
values are not greater than expected.  As dir_count and size were
incorrectly defined as signed ints, this can lead to corrupted values
appearing as negative which are not trapped.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

Patch "Squashfs: sanity check information from disk" from
Dan Carpenter adds a missing check for corruption in the
"size" field while reading the directory index from disk.

It, however, sets err to -EINVAL, this value is not used later, and
so setting it is completely redundant.  So remove it.

Errors in reading the index are deliberately non-fatal.  If we
get an error in reading the index we just return the part of the
index we have managed to read - the index isn't essential,
just quicker.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

Merged the two for loops. We might get a little gain by overlapping
wait_on_bh and the memcpy operations.
Signed-off-by: NManish Sharma <manishrma@gmail.com>
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

We read the size of the name from the disk, but a larger name than
expected would cause memory corruption.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

Signed-off-by: NAl Viro <viro@zeniv.linux.org.uk>

Somehow I failed to add the MODULE_ALIAS_FS for cifs, hostfs, hpfs,
squashfs, and udf despite what I thought were my careful checks :(

Add them now.
Signed-off-by: N"Eric W. Biederman" <ebiederm@xmission.com>

Signed-off-by: NAl Viro <viro@zeniv.linux.org.uk>

There's no reason to call rcu_barrier() on every
deactivate_locked_super().  We only need to make sure that all delayed rcu
free inodes are flushed before we destroy related cache.

Removing rcu_barrier() from deactivate_locked_super() affects some fast
paths.  E.g.  on my machine exit_group() of a last process in IPC
namespace takes 0.07538s.  rcu_barrier() takes 0.05188s of that time.
Signed-off-by: NKirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NAl Viro <viro@zeniv.linux.org.uk>

Cc: Phillip Lougher <phillip@squashfs.org.uk>
Acked-by: NSerge Hallyn <serge.hallyn@canonical.com>
Signed-off-by: NEric W. Biederman <ebiederm@xmission.com>

Just the flags; only NFS cares even about that, but there are
legitimate uses for such argument.  And getting rid of that
completely would require splitting ->lookup() into a couple
of methods (at least), so let's leave that alone for now...
Signed-off-by: NAl Viro <viro@zeniv.linux.org.uk>

Signed-off-by: NAl Viro <viro@zeniv.linux.org.uk>

Signed-off-by: NCong Wang <amwang@redhat.com>

Squashfs currently has a sanity check for block_size less than or
equal to the maximum block_size (1 Mbyte).  This catches some
superblock corruption, but obviously with a block_size maximum
of 1 Mbyte there's 7 correct values (4K, 8K, 16K, 32K, ... etc) and
a lot of incorrect values which are not caught by this check.

The Squashfs superblock, however, has both a block_size and
a block_log (2^block_log == block_size).  Checking that the block_size
matches the block_log is a much more robust check.  Corruption of the
superblock is unlikely to produce values which match, and it also
ensures the block_size is an exact power of two.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

One off error in the f_pos check.  If f_pos is 3 or less don't
bother reading the index because we're at the start of the
directory, and we obviously already know where that is on disk.

This eliminates an unnecessary read.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

Most of these were never used by the kernel code, but belong to
the time when the header file was used by both the kernel code
and the user space tools.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

Comment was written when Squashfs only supported zlib compression.
This comment is now misleading given Squashfs supports other
compression algorithms.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

Its better to use defined name instead of constant
Signed-off-by: NAjeet Yadav <ajeet.yadav.77@gmail.com>
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

Seeing that just about every destructor got that INIT_LIST_HEAD() copied into
it, there is no point whatsoever keeping this INIT_LIST_HEAD in inode_init_once();
the cost of taking it into inode_init_always() will be negligible for pipes
and sockets and negative for everything else.  Not to mention the removal of
boilerplate code from ->destroy_inode() instances...
Signed-off-by: NAl Viro <viro@zeniv.linux.org.uk>

The le64_to_cpu() forces the calculation to be unsigned, with
the effect that it can underflow leading to an incorrect large
value.

This bug only triggers in rare(ish) circumstances, an empty file
encoded as an extended regular file or a completely sparse file.
Normally empty files are encoded as a regular file rather than as
an extended regular file (and the regular file i_blocks calculation
doesn't have this bug).  To save space regular file inodes are
optimised to encode the most commonly occurring files.  Less
common regular files are encoded using extended regular file inodes
which contain extra information.

Empty files with nlinks greater than 1, and or empty files
with extended attributes are encoded using extended regular file
inodes and they will hit this bug.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

A Squashfs filesystem containing nothing but an empty directory,
although unusual and ultimately pointless, is still valid.

The directory_table >= next_table sanity check rejects these
filesystems as invalid because the directory_table is empty and
equal to next_table.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

squashfs_cache_get() iterates over all entries to search for
 block its looking for. Often get() / put() are called for
 same block.

If we cache the current entry index, then we can optimise the
subsequent *_get() calls.
Signed-off-by: NAjeet Yadav <ajeet.yadav.77@gmail.com>
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

squashfs_read_metadata forgets to release the cache block if
an error has occurred.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

This commit adds an option to set the device block size used to 4K.

By default Squashfs sets the device block size (sb_min_blocksize) to 1K
or the smallest block size supported by the block device (if larger).
This, because blocks are packed together and unaligned in Squashfs,
should reduce latency.

This, however, gives poor performance on MTD NAND devices where
the optimal I/O size is 4K (even though the devices can support
smaller block sizes).

Using a 4K device block size may also improve overall I/O
performance for some file access patterns (e.g. sequential
accesses of files in filesystem order) on all media.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

Replace remaining direct i_nlink updates with a new set_nlink()
updater function.
Signed-off-by: NMiklos Szeredi <mszeredi@suse.cz>
Tested-by: NToshiyuki Okajima <toshi.okajima@jp.fujitsu.com>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

There are numerous broken references to Documentation files (in other
Documentation files, in comments, etc.). These broken references are
caused by typo's in the references, and by renames or removals of the
Documentation files. Some broken references are simply odd.

Fix these broken references, sometimes by dropping the irrelevant text
they were part of.
Signed-off-by: NPaul Bolle <pebolle@tiscali.nl>
Signed-off-by: NJiri Kosina <jkosina@suse.cz>

Squashfs now supports XZ and LZO compression in addition to ZLIB.
As such it no longer makes sense to always include ZLIB support.
In particular embedded systems may only use LZO or XZ compression, and
the ability to exclude ZLIB support will reduce kernel size.
Signed-off-by: NPhillip Lougher <phillip@squashfs.org.uk>

... and simplify the living hell out of callers
Signed-off-by: NAl Viro <viro@zeniv.linux.org.uk>

d_splice_alias(NULL, dentry) is equivalent to d_add(dentry, NULL), NULL
so no need for that if (inode) ... in there (or ERR_PTR(0), for that
matter)
Signed-off-by: NAl Viro <viro@zeniv.linux.org.uk>

le64 values should be swapped when accessing on
big-endian systems.
Signed-off-by: NPhillip Lougher <phillip@lougher.demon.co.uk>

My existing email address may stop working in a month or two, so update
email to one that will continue working.
Signed-off-by: NPhillip Lougher <phillip@lougher.demon.co.uk>