In media spaces, video is often stored in a frame-per-file format. When
dealing with uncompressed realtime HD video streams in this format, it is
crucial that files do not get fragmented and that multiple files a placed
contiguously on disk.

When multiple streams are being ingested and played out at the same time,
it is critical that the filesystem does not cross the streams and
interleave them together as this creates seek and readahead cache miss
latency and prevents both ingest and playout from meeting frame rate
targets.

This patch set creates a "stream of files" concept into the allocator to
place all the data from a single stream contiguously on disk so that RAID
array readahead can be used effectively. Each additional stream gets
placed in different allocation groups within the filesystem, thereby
ensuring that we don't cross any streams. When an AG fills up, we select a
new AG for the stream that is not in use.

The core of the functionality is the stream tracking - each inode that we
create in a directory needs to be associated with the directories' stream.
Hence every time we create a file, we look up the directories' stream
object and associate the new file with that object.

Once we have a stream object for a file, we use the AG that the stream
object point to for allocations. If we can't allocate in that AG (e.g. it
is full) we move the entire stream to another AG. Other inodes in the same
stream are moved to the new AG on their next allocation (i.e. lazy
update).

Stream objects are kept in a cache and hold a reference on the inode.
Hence the inode cannot be reclaimed while there is an outstanding stream
reference. This means that on unlink we need to remove the stream
association and we also need to flush all the associations on certain
events that want to reclaim all unreferenced inodes (e.g. filesystem
freeze).

SGI-PV: 964469
SGI-Modid: xfs-linux-melb:xfs-kern:29096a
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NBarry Naujok <bnaujok@sgi.com>
Signed-off-by: NDonald Douwsma <donaldd@sgi.com>
Signed-off-by: NChristoph Hellwig <hch@infradead.org>
Signed-off-by: NTim Shimmin <tes@sgi.com>
Signed-off-by: NVlad Apostolov <vapo@sgi.com>

A check for file_count is always a bad idea. Linux has the ->release
method to deal with cleanups on last close and ->flush is only for the
very rare case where we want to perform an operation on every drop of a
reference to a file struct.

This patch gets rid of vop_close and surrounding code in favour of simply
doing the page flushing from ->release.

SGI-PV: 966562
SGI-Modid: xfs-linux-melb:xfs-kern:28952a
Signed-off-by: NChristoph Hellwig <hch@infradead.org>
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

SGI-PV: 966502
SGI-Modid: xfs-linux-melb:xfs-kern:28943a
Signed-off-by: NJesper Juhl <jesper.juhl@gmail.com>
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

SGI-PV: 966004
SGI-Modid: xfs-linux-melb:xfs-kern:28866a
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NChristoph Hellwig <hch@infradead.org>
Signed-off-by: NTim Shimmin <tes@sgi.com>

If hole punching at EOF is done as two steps (i.e. truncate then extend)
the file is in a transient state between the two steps where an
application can see the incorrect file size. Punching a hole to EOF needs
to be treated in teh same way as all other hole punching cases so that the
file size is never seen to change.

SGI-PV: 962012
SGI-Modid: xfs-linux-melb:xfs-kern:28641a
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NVlad Apostolov <vapo@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

They can use generic_file_splice_read() instead. Since sys_sendfile() now
prefers that, there should be no change in behaviour.
Signed-off-by: NJens Axboe <jens.axboe@oracle.com>

SGI-PV: 963965
SGI-Modid: xfs-linux-melb:xfs-kern:28485a
Signed-off-by: NLachlan McIlroy <lachlan@sgi.com>
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

SGI-PV: 907752
SGI-Modid: xfs-linux-melb:xfs-kern:28344a
Signed-off-by: NTim Shimmin <tes@sgi.com>
Signed-off-by: NLachlan McIlroy <lachlan@sgi.com>
Signed-off-by: NEric Sandeen <sandeen@sandeen.net>

The problem that has been addressed is that of synchronising updates of
the file size with writes that extend a file. Without the fix the update
of a file's size, as a result of a write beyond eof, is independent of
when the cached data is flushed to disk. Often the file size update would
be written to the filesystem log before the data is flushed to disk. When
a system crashes between these two events and the filesystem log is
replayed on mount the file's size will be set but since the contents never
made it to disk the file is full of holes. If some of the cached data was
flushed to disk then it may just be a section of the file at the end that
has holes.

There are existing fixes to help alleviate this problem, particularly in
the case where a file has been truncated, that force cached data to be
flushed to disk when the file is closed. If the system crashes while the
file(s) are still open then this flushing will never occur.

The fix that we have implemented is to introduce a second file size,
called the in-memory file size, that represents the current file size as
viewed by the user. The existing file size, called the on-disk file size,
is the one that get's written to the filesystem log and we only update it
when it is safe to do so. When we write to a file beyond eof we only
update the in- memory file size in the write operation. Later when the I/O
operation, that flushes the cached data to disk completes, an I/O
completion routine will update the on-disk file size. The on-disk file
size will be updated to the maximum offset of the I/O or to the value of
the in-memory file size if the I/O includes eof.

SGI-PV: 958522
SGI-Modid: xfs-linux-melb:xfs-kern:28322a
Signed-off-by: NLachlan McIlroy <lachlan@sgi.com>
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

This patch handles error return values in fs_flush_pages and
fs_flushinval_pages. It changes the prototype of fs_flushinval_pages so we
can propogate the errors and handle them at higher layers. I also modified
xfs_itruncate_start so that it could propogate the error further.

SGI-PV: 961990
SGI-Modid: xfs-linux-melb:xfs-kern:28231a
Signed-off-by: NLachlan McIlroy <lachlan@sgi.com>
Signed-off-by: NStewart Smith <stewart@flamingspork.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

Patch provided by Eric Sandeen.

SGI-PV: 961695
SGI-Modid: xfs-linux-melb:xfs-kern:28205a
Signed-off-by: NEric Sandeen <sandeen@sandeen.net>
Signed-off-by: NLachlan McIlroy <lachlan@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

NULL.

Patch provided by Eric Sandeen.

SGI-PV: 961693
SGI-Modid: xfs-linux-melb:xfs-kern:28199a
Signed-off-by: NEric Sandeen <sandeen@sandeen.net>
Signed-off-by: NLachlan McIlroy <lachlan@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

xfs_mac.h and xfs_cap.h provide definitions and macros that aren't used
anywhere in XFS at all. They are left-overs from "to be implement at some
point in the future" functionality that Irix XFS has. If this
functionality ever goes into Linux, it will be provided at a different
layer, most likely through the security hooks in the kernel so we will
never need this functionality in XFS.

Patch provided by Eric Sandeen (sandeen@sandeen.net).

SGI-PV: 960895
SGI-Modid: xfs-linux-melb:xfs-kern:28036a
Signed-off-by: NEric Sandeen <sandeen@sandeen.net>
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

The firstblock argument to xfs_bmap_finish is not used by that function.
Remove it and cleanup the code a bit.

Patch provided by Eric Sandeen.

SGI-PV: 960196
SGI-Modid: xfs-linux-melb:xfs-kern:28034a
Signed-off-by: NEric Sandeen <sandeen@sandeen.net>
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

The previous fixes for the use after free in xfs_iunpin left a nasty log
deadlock when xfslogd unpinned the inode and dropped the last reference to
the inode. the ->clear_inode() method can issue transactions, and if the
log was full, the transaction could push on the log and get stuck trying
to push the inode it was currently unpinning.

To fix this, we provide xfs_iunpin a guarantee that it will always have a
valid xfs_inode <-> linux inode link or a particular flag will be set on
the inode. We then use log forces during lookup to ensure transactions are
completed before we recycle the inode. This ensures that xfs_iunpin will
never use the linux inode after it is being freed, and any lookup on an
inode on the reclaim list will wait until it is safe to attach a new linux
inode to the xfs inode.

SGI-PV: 956832
SGI-Modid: xfs-linux-melb:xfs-kern:27359a
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NShailendra Tripathi <stripathi@agami.com>
Signed-off-by: NTakenori Nagano <t-nagano@ah.jp.nec.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

SGI-PV: 956832
SGI-Modid: xfs-linux-melb:xfs-kern:27358a
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NNathan Scott <nscott@aconex.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

SGI-PV: 957004
SGI-Modid: xfs-linux-melb:xfs-kern:27231a
Signed-off-by: NVlad Apostolov <vapo@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

The previous attempts to fix the linux inode use-after-free in xfs_iunpin
simply made the problem harder to hit. We actually need complete exclusion
between xfs_reclaim and xfs_iunpin, as well as ensuring that the i_flags
are consistent during both of these functions. Introduce a new spinlock
for exclusion and the i_flags, and fix up xfs_iunpin to use igrab before
marking the inode dirty.

SGI-PV: 952967
SGI-Modid: xfs-linux-melb:xfs-kern:26964a
Signed-off-by: NDavid Chinner <dgc@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

SGI-PV: 953687
SGI-Modid: xfs-linux-melb:xfs-kern:26639a
Signed-off-by: NVlad Apostolov <vapo@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

SGI-PV: 954580
SGI-Modid: xfs-linux-melb:xfs-kern:26551a
Signed-off-by: NNathan Scott <nathans@sgi.com>
Signed-off-by: NTim Shimmin <tes@sgi.com>

SGI-PV: 904196
SGI-Modid: xfs-linux-melb:xfs-kern:26293a
Signed-off-by: NAlexey Dobriyan <adobriyan@gmail.com>
Signed-off-by: NNathan Scott <nathans@sgi.com>

pure bloat.

SGI-PV: 952969
SGI-Modid: xfs-linux-melb:xfs-kern:26251a
Signed-off-by: NNathan Scott <nathans@sgi.com>

porters.

SGI-PV: 953338
SGI-Modid: xfs-linux-melb:xfs-kern:26108a
Signed-off-by: NNathan Scott <nathans@sgi.com>

SGI-PV: 953338
SGI-Modid: xfs-linux-melb:xfs-kern:26107a
Signed-off-by: NNathan Scott <nathans@sgi.com>

truncate down followed by delayed allocation (buffered writes) - worst
case scenario for the notorious NULL files problem.  This reduces the
window where we are exposed to that problem significantly.

SGI-PV: 917976
SGI-Modid: xfs-linux-melb:xfs-kern:26100a
Signed-off-by: NNathan Scott <nathans@sgi.com>

shutdown vop flags consistent with sync vop flags declarations too.

SGI-PV: 939911
SGI-Modid: xfs-linux-melb:xfs-kern:26096a
Signed-off-by: NNathan Scott <nathans@sgi.com>

SGI-PV: 953061
SGI-Modid: xfs-linux-melb:xfs-kern:25986a
Signed-off-by: NBarry Naujok <bnaujok@sgi.com>
Signed-off-by: NNathan Scott <nathans@sgi.com>

ATTR_NOLOCK flag, but this was split off some time ago, as ATTR_DMI needed
to be used separately.	Two asserts were added to guard correctness of the
code during the transition.  These are no longer required.

SGI-PV: 952145
SGI-Modid: xfs-linux-melb:xfs-kern:209633a
Signed-off-by: NOlaf Weber <olaf@sgi.com>
Signed-off-by: NNathan Scott <nathans@sgi.com>

the range spanned by modifications to the in-core extent map.  Add
XFS_BUNMAPI() and XFS_SWAP_EXTENTS() macros that call xfs_bunmapi() and
xfs_swap_extents() via the ioops vector. Change all calls that may modify
the in-core extent map for the data fork to go through the ioops vector. 
This allows a cache of extent map data to be kept in sync.

SGI-PV: 947615
SGI-Modid: xfs-linux-melb:xfs-kern:209226a
Signed-off-by: NOlaf Weber <olaf@sgi.com>
Signed-off-by: NNathan Scott <nathans@sgi.com>

SGI-PV: 951636
SGI-Modid: xfs-linux-melb:xfs-kern:25811a
Signed-off-by: NNathan Scott <nathans@sgi.com>

Signed-off-by: NNathan Scott <nathans@sgi.com>

these typos.

SGI-PV: 904196
SGI-Modid: xfs-linux-melb:xfs-kern:25539a
Signed-off-by: NNathan Scott <nathans@sgi.com>

Use ARRAY_SIZE macro instead of sizeof(x)/sizeof(x[0]) and remove a
duplicate of ARRAY_SIZE.  Some trailing whitespaces are also deleted.
Signed-off-by: NTobias Klauser <tklauser@nuerscht.ch>
Cc: David Howells <dhowells@redhat.com>
Cc: Dave Kleikamp <shaggy@austin.ibm.com>
Acked-by: NTrond Myklebust <trond.myklebust@fys.uio.no>
Cc: Neil Brown <neilb@cse.unsw.edu.au>
Cc: Chris Mason <mason@suse.com>
Cc: Jeff Mahoney <jeffm@suse.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Nathan Scott <nathans@sgi.com>
Signed-off-by: NAndrew Morton <akpm@osdl.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Signed-off-by: NNathan Scott <nathans@sgi.com>

conversion and concurrent truncate operations. Use vn_iowait to wait for
the completion of any pending DIOs. Since the truncate requires exclusive
IOLOCK, so this blocks any further DIO operations since DIO write also
needs exclusive IOBLOCK. This serves as a barrier and prevent any
potential starvation.

SGI-PV: 947420
SGI-Modid: xfs-linux-melb:xfs-kern:208088a
Signed-off-by: NYingping Lu <yingping@sgi.com>
Signed-off-by: NNathan Scott <nathans@sgi.com>

SGI-PV: 947038
SGI-Modid: xfs-linux-melb:xfs-kern:25474a
Signed-off-by: NNathan Scott <nathans@sgi.com>

This patch removes pointless parentheses from return statements.
Signed-off-by: NJesper Juhl <juhl-lkml@dif.dk>
Signed-off-by: NAdrian Bunk <bunk@stusta.de>

fs: Use <linux/capability.h> where capable() is used.
Signed-off-by: NRandy Dunlap <rdunlap@xenotime.net>
Acked-by: NTim Schmielau <tim@physik3.uni-rostock.de>
Signed-off-by: NAndrew Morton <akpm@osdl.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

SGI-PV: 946679
SGI-Modid: xfs-linux-melb:xfs-kern:24899a
Signed-off-by: NNathan Scott <nathans@sgi.com>

SGI-PV: 947206
SGI-Modid: xfs-linux-melb:xfs-kern:203960a
Signed-off-by: NChristoph Hellwig <hch@sgi.com>
Signed-off-by: NNathan Scott <nathans@sgi.com>