There was a semi-colon missing and it broke the compile.
Signed-off-by: NDan Carpenter <error27@gmail.com>
Cc: Dmitry Torokhov <dmitry.torokhov@gmail.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Minor formatting fixup since the information which core was associated
with the MCE is not always valid.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Building for X86_32 produces shift count warnings, so use BIT_64() to
eliminate the warnings.

drivers/edac/mce_amd.c:778: warning: left shift count >= width of type
drivers/edac/mce_amd.c:778: warning: left shift count >= width of type
Signed-off-by: NRandy Dunlap <randy.dunlap@oracle.com>
Cc: Doug Thompson <dougthompson@xmission.com>
Cc: bluesmoke-devel@lists.sourceforge.net
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Now that everything is inplace, enable MCE decoding on F15h. Make
initcall routine a bit more readable.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

F15h adds a sixth MCE bank: adjust bank number check in the injection
code.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Shorten up MCi_STATUS flags and add BD's new deferred and poison types.
Also, simplify formatting.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Make macro names shorter thus making code shorter and more clear.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Add decoder for FP MCEs.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Integrate the single FIROB signature into an expanded table along with
the new BD MCE types.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

by (almost) reusing the F10h one since the signatures are the same.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

F15h BD doesn't generate LS MCEs so warn about it.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

MCE bank 2 is redefined from a BU to a CU (Combined Unit) bank on F15h.
Add a decoder function for CU MCEs.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Add support for decoding F15h IC MCEs.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Add a decoder for F15h DC MCEs to support the new types of DC MCEs
introduced by the BD microarchitecture.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

F15h enlarges the extended error code of an MCE to a 5-bit field
(MCi_STATUS[20:16]). Add a mask variable which default 0xf is overridden
on F15h.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

K8 does not allow for an atomic RMW to a cacheline as F10h does so
disable the error injection interface for it.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Make the ->{get|set}_sdram_scrub_rate return the actual scrub rate
bandwidth it succeeded setting and remove superfluous arg pointer used
for that. A negative value returned still means that an error occurred
while setting the scrubrate. Document this for future reference.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Now that all prerequisites are in place, drop the two-stage driver
instances initialization in favor of the following simple init sequence:

1. Probe PCI device: we only test ECC capabilities here and if none exit
early.

2. If the hw supports ECC and it is/can be enabled, we init the per-node
instance.

Remove "amd64_" prefix from static functions touched, while at it.

There actually should be no visible functional change resulting from
this patch.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Rework the code to check the hardware ECC capabilities at PCI probing
time. We do all further initialization only if we actually can/have ECC
enabled.

While at it:
0. Fix function naming.
1. Simplify/clarify debug output.
2. Remove amd64_ prefix from the static functions
3. Reorganize code.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

This is in preparation for the init path reorganization where we want
only to

1) test whether a particular node supports ECC
2) can it be enabled

and only then do the necessary allocation/initialization. For that,
we need to decouple the ECC settings of the node from the instance's
descriptor.

The should be no functional change introduced by this patch.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

PCI ECS is being enabled by default since 2.6.26 on AMD so this code is
just superfluous now, remove it.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

AMD_NB pulls in the dependency on PCI. Clarify/fix help text while at it.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Remove static allocation in favor of dynamically allocating space for as
many driver instances as northbridges present on the system.

There should be no functional change resulting from this patch.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Add a macro per printk level, shorten up error messages. Add relevant
information to KERN_INFO level. No functional change.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Rename variables representing PCI devices to their BKDG names for faster
search and shorter, clearer code.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Move the remaining per-family init code into the proper place and
simplify the rest of the initialization. Reorganize error handling in
amd64_init_one_instance().
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Shorten code and clarify comments, return proper -E* values on error.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Concentrate CPU family detection in the per-family init function.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

Run a per-family init function which does all the settings based on
the family this driver instance is running on. Move the scrubrate
calculation in it and simplify code.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

... instead of computing it needlessly again.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

F11h doesn't support DRAM ECC so whack it away.
Signed-off-by: NBorislav Petkov <borislav.petkov@amd.com>

The problem that this patch aims to fix is vfsmount refcounting scalability.
We need to take a reference on the vfsmount for every successful path lookup,
which often go to the same mount point.

The fundamental difficulty is that a "simple" reference count can never be made
scalable, because any time a reference is dropped, we must check whether that
was the last reference. To do that requires communication with all other CPUs
that may have taken a reference count.

We can make refcounts more scalable in a couple of ways, involving keeping
distributed counters, and checking for the global-zero condition less
frequently.

- check the global sum once every interval (this will delay zero detection
  for some interval, so it's probably a showstopper for vfsmounts).

- keep a local count and only taking the global sum when local reaches 0 (this
  is difficult for vfsmounts, because we can't hold preempt off for the life of
  a reference, so a counter would need to be per-thread or tied strongly to a
  particular CPU which requires more locking).

- keep a local difference of increments and decrements, which allows us to sum
  the total difference and hence find the refcount when summing all CPUs. Then,
  keep a single integer "long" refcount for slow and long lasting references,
  and only take the global sum of local counters when the long refcount is 0.

This last scheme is what I implemented here. Attached mounts and process root
and working directory references are "long" references, and everything else is
a short reference.

This allows scalable vfsmount references during path walking over mounted
subtrees and unattached (lazy umounted) mounts with processes still running
in them.

This results in one fewer atomic op in the fastpath: mntget is now just a
per-CPU inc, rather than an atomic inc; and mntput just requires a spinlock
and non-atomic decrement in the common case. However code is otherwise bigger
and heavier, so single threaded performance is basically a wash.
Signed-off-by: NNick Piggin <npiggin@kernel.dk>

Signed-off-by: NNick Piggin <npiggin@kernel.dk>

Require filesystems be aware of .d_revalidate being called in rcu-walk
mode (nd->flags & LOOKUP_RCU). For now do a simple push down, returning
-ECHILD from all implementations.
Signed-off-by: NNick Piggin <npiggin@kernel.dk>

Reduce some branches and memory accesses in dcache lookup by adding dentry
flags to indicate common d_ops are set, rather than having to check them.
This saves a pointer memory access (dentry->d_op) in common path lookup
situations, and saves another pointer load and branch in cases where we
have d_op but not the particular operation.

Patched with:

git grep -E '[.>]([[:space:]])*d_op([[:space:]])*=' | xargs sed -e 's/\([^\t ]*\)->d_op = \(.*\);/d_set_d_op(\1, \2);/' -e 's/\([^\t ]*\)\.d_op = \(.*\);/d_set_d_op(\&\1, \2);/' -i
Signed-off-by: NNick Piggin <npiggin@kernel.dk>

RCU free the struct inode. This will allow:

- Subsequent store-free path walking patch. The inode must be consulted for
  permissions when walking, so an RCU inode reference is a must.
- sb_inode_list_lock to be moved inside i_lock because sb list walkers who want
  to take i_lock no longer need to take sb_inode_list_lock to walk the list in
  the first place. This will simplify and optimize locking.
- Could remove some nested trylock loops in dcache code
- Could potentially simplify things a bit in VM land. Do not need to take the
  page lock to follow page->mapping.

The downsides of this is the performance cost of using RCU. In a simple
creat/unlink microbenchmark, performance drops by about 10% due to inability to
reuse cache-hot slab objects. As iterations increase and RCU freeing starts
kicking over, this increases to about 20%.

In cases where inode lifetimes are longer (ie. many inodes may be allocated
during the average life span of a single inode), a lot of this cache reuse is
not applicable, so the regression caused by this patch is smaller.

The cache-hot regression could largely be avoided by using SLAB_DESTROY_BY_RCU,
however this adds some complexity to list walking and store-free path walking,
so I prefer to implement this at a later date, if it is shown to be a win in
real situations. I haven't found a regression in any non-micro benchmark so I
doubt it will be a problem.
Signed-off-by: NNick Piggin <npiggin@kernel.dk>

dget_locked was a shortcut to avoid the lazy lru manipulation when we already
held dcache_lock (lru manipulation was relatively cheap at that point).
However, how that the lru lock is an innermost one, we never hold it at any
caller, so the lock cost can now be avoided. We already have well working lazy
dcache LRU, so it should be fine to defer LRU manipulations to scan time.
Signed-off-by: NNick Piggin <npiggin@kernel.dk>

dcache_lock no longer protects anything. remove it.
Signed-off-by: NNick Piggin <npiggin@kernel.dk>

The remaining usages for dcache_lock is to allow atomic, multi-step read-side
operations over the directory tree by excluding modifications to the tree.
Also, to walk in the leaf->root direction in the tree where we don't have
a natural d_lock ordering.

This could be accomplished by taking every d_lock, but this would mean a
huge number of locks and actually gets very tricky.

Solve this instead by using the rename seqlock for multi-step read-side
operations, retry in case of a rename so we don't walk up the wrong parent.
Concurrent dentry insertions are not serialised against.  Concurrent deletes
are tricky when walking up the directory: our parent might have been deleted
when dropping locks so also need to check and retry for that.

We can also use the rename lock in cases where livelock is a worry (and it
is introduced in subsequent patch).
Signed-off-by: NNick Piggin <npiggin@kernel.dk>

Protect d_subdirs and d_child with d_lock, except in filesystems that aren't
using dcache_lock for these anyway (eg. using i_mutex).

Note: if we change the locking rule in future so that ->d_child protection is
provided only with ->d_parent->d_lock, it may allow us to reduce some locking.
But it would be an exception to an otherwise regular locking scheme, so we'd
have to see some good results. Probably not worthwhile.
Signed-off-by: NNick Piggin <npiggin@kernel.dk>