None of the files touched here are modules, and they are not
exporting any symbols either -- so there is no need to be including
the module.h.  Builds of all the files remains successful.

Even kernel/module.c does not need to include it, since it includes
linux/moduleloader.h instead.
Signed-off-by: NPaul Gortmaker <paul.gortmaker@windriver.com>

They are getting it through device.h --> module.h path, but we want
to clean that up.  This is a sample of what will happen if we don't:

  pseries/iommu.c: In function 'tce_build_pSeriesLP':
  pseries/iommu.c:136: error: implicit declaration of function 'show_stack'

  pseries/eeh.c: In function 'eeh_token_to_phys':
  pseries/eeh.c:359: error: 'init_mm' undeclared (first use in this function)

  pseries/eeh_event.c: In function 'eeh_event_handler':
  pseries/eeh_event.c:63: error: implicit declaration of function 'daemonize'
  pseries/eeh_event.c:64: error: implicit declaration of function 'set_current_state'
  pseries/eeh_event.c:64: error: 'TASK_INTERRUPTIBLE' undeclared (first use in this function)
  pseries/eeh_event.c:64: error: (Each undeclared identifier is reported only once
  pseries/eeh_event.c:64: error: for each function it appears in.)
  pseries/eeh_event.c: In function 'eeh_thread_launcher':
  pseries/eeh_event.c:109: error: 'CLONE_KERNEL' undeclared (first use in this function)

  hotplug-cpu.c: In function 'pseries_mach_cpu_die':
  hotplug-cpu.c:115: error: implicit declaration of function 'idle_task_exit'

  kernel/swsusp_64.c: In function 'do_after_copyback':
  kernel/swsusp_64.c:17: error: implicit declaration of function 'touch_softlockup_watchdog'

  cell/spufs/context.c: In function 'alloc_spu_context':
  cell/spufs/context.c:60: error: implicit declaration of function 'get_task_mm'
  cell/spufs/context.c:60: warning: assignment makes pointer from integer without a cast
  cell/spufs/context.c: In function 'spu_forget':
  cell/spufs/context.c:127: error: implicit declaration of function 'mmput'

  pasemi/dma_lib.c: In function 'pasemi_dma_stop_chan':
  pasemi/dma_lib.c:332: error: implicit declaration of function 'cond_resched'

  sysdev/fsl_lbc.c: In function 'fsl_lbc_ctrl_irq':
  sysdev/fsl_lbc.c:247: error: 'TASK_NORMAL' undeclared (first use in this function)

Add in sched.h so these get the definitions they are looking for.
Signed-off-by: NPaul Gortmaker <paul.gortmaker@windriver.com>

This allows us to move duplicated code in <asm/atomic.h>
(atomic_inc_not_zero() for now) to <linux/atomic.h>
Signed-off-by: NArun Sharma <asharma@fb.com>
Reviewed-by: NEric Dumazet <eric.dumazet@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: David Miller <davem@davemloft.net>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Acked-by: NMike Frysinger <vapier@gentoo.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

I wrote sputrace before generic tracing infrastrucure was available.
Now that we have the generic event tracer we can convert it over and
remove a lot of code:

  8 files changed, 45 insertions(+), 285 deletions(-)

To use it make sure CONFIG_EVENT_TRACING is enabled and then enable
the spufs trace channel by

  echo 1 > /sys/kernel/debug/tracing/events/spufs/spufs_context/enable

and then read the trace records using e.g.

  cat /sys/kernel/debug/tracing/trace
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

spuctx_switch_state() warns if ktime goes backwards, but it
sometimes compares an uninitialized value, which showed that
the data was unreliable when we actually saw the warning.

Initialize it to the current time in order to get correct data.
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Use a set of #defines for the size of context mappings, instead of
magic numbers.
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>

The sputrace module contained a trace entry for spu_acquire_saved, but
this marker was not placed anywhere. Fix this by adding a marker to the
routine.
Signed-off-by: NJulio M. Merino Vidal <jmerino@ac.upc.edu>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>

The sputrace module contained a reference to a marker for
destroy_spu_context, but this marker did not appear in the code. Fix
this by adding a marker in the function.
Signed-off-by: NJulio M. Merino Vidal <jmerino@ac.upc.edu>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>

There are userspace instrumentation tools that need to monitor spu
context switches. This patch adds a new file called 'switch_log' to
each spufs context directory that can be used to monitor the context
switches.

Context switch in/out and exit from spu_run are monitored after the
file was first opened and can be read from it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>

At present, we can hit the BUG_ON in __spu_update_sched_info by reading
the regs file of a context between two calls to spu_run. The
spu_release_saved called by spufs_regs_read() is resulting in the (now
non-runnable) context being placed back on the run queue, so the next
call to spu_run ends up in the bug condition.

This change uses the SPU_SCHED_SPU_RUN flag to only reschedule a context
if it's still in spu_run().
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>

At present, we have a situation where a context with no owner is
re-scheduled by spu_forget:

	Thread 1: reading regs file	Thread 2: context owner

					spu_forget()
						- ctx->owner = NULL
						- set SPU_SCHED_WAS_ACTIVE

	spu_acquire_saved()
	- context is in saved state

	spu_release_saved()
	- SPU_SCHED_WAS_ACTIVE is set,
	  so spu_activate() the context,
	  which now has no owner

In spu_forget(), we shouldn't be requesting a re-schedule by setting
SPU_SCHED_WAS_ACTIVE. This change removes the set_bit in spu_forget(),
so that spu_release_saved() doesn't reinsert this destroyed context on
to the run queue.
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>

This removes an OProfile dependency on the spufs module.  This
dependency was causing a problem for multiplatform systems that are
built with support for Oprofile on Cell but try to load the oprofile
module on a non-Cell system.
Signed-off-by: NBob Nelson <rrnelson@us.ibm.com>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>
Acked-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

Make most places that use spu_acquire/spu_acquire_saved interruptible,
this allows getting out of the spufs code when e.g. pressing ctrl+c.
There are a few places where we get called e.g. from spufs teardown
routines were we can't simply err out so these are left with a comment.
For now I've also not touched the poll routines because it's open what
libspe would expect in terms of interrupted system calls.
Acked-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

Change spufs_spu_run so that the context is queued directly to the
scheduler and the controlling thread advances directly to spufs_wait()
for spe errors and exceptions.

nosched contexts are treated the same as before.

Fixes from Christoph Hellwig <hch@lst.de>
Signed-off-by: NLuke Browning <lukebr@linux.vnet.ibm.com>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

This change disables the logic that faults-in spu contexts under the
covers from the page fault handler.  When a fault requires a runnable
context, the handler will block until the context is scheduled by
other means.
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

Currently we calculate the first timeslice for every context
incorrectly - alloc_spu_context calls spu_set_timeslice before we set
ctx->prio so we always calculate the longest possible timeslice for the
lowest possible priority.

This patch makes sure to update the schedule-related fields before
calculating the timeslice and also makes sure we update the timeslice for
a non-running context when entering spu_run so a priority change affects
the context as soon as possible.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

From: Maynard Johnson <mpjohn@us.ibm.com>

This patch updates the existing arch/powerpc/oprofile/op_model_cell.c
to add in the SPU profiling capabilities.  In addition, a 'cell' subdirectory
was added to arch/powerpc/oprofile to hold Cell-specific SPU profiling code.
Exports spu_set_profile_private_kref and spu_get_profile_private_kref which
are used by OProfile to store private profile information in spufs data
structures.

Also incorporated several fixes from other patches (rrn).  Check pointer
returned from kzalloc.  Eliminated unnecessary cast.  Better error
handling and cleanup in the related area.  64-bit unsigned long parameter
was being demoted to 32-bit unsigned int and eventually promoted back to
unsigned long.
Signed-off-by: NCarl Love <carll@us.ibm.com>
Signed-off-by: NMaynard Johnson <mpjohn@us.ibm.com>
Signed-off-by: NBob Nelson <rrnelson@us.ibm.com>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>
Acked-by: NPaul Mackerras <paulus@samba.org>

This patch adds support for additional flags at spu_create, which relate
to the establishment of affinity between contexts and contexts to memory.
A fourth, optional, parameter is supported. This parameter represent
a affinity neighbor of the context being created, and is used when defining
SPU-SPU affinity.
Affinity is represented as a doubly linked list of spu_contexts.
Signed-off-by: NAndre Detsch <adetsch@br.ibm.com>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

Currently a process is removed from the physical spu when spu_acquire_saved
is saved but never put back.  This patch adds a new spu_release_saved
that is to be paired with spu_acquire_saved and put the process back if
it has been in RUNNABLE state before.

Niether Jeremy not be are entirely happy about this exact patch because
it adds another spu_activate call outside of the owner thread, but I
feel this is the best short-term fix we can come up with.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

This patch exports per-context statistics in spufs as long as spu
statistics in sysfs.

It was formed by merging:
"spufs: add spu stats in sysfs"   From: Christoph Hellwig
"spufs: add stat file to spufs"   From: Christoph Hellwig
"spufs: fix libassist accounting" From: Jeremy Kerr
"spusched: fix spu utilization statistics" From: Luke Browning
And some adjustments by myself, after suggestions on cbe-oss-dev.

Having separate patches was making the review process harder
than it should, as we end up integrating spus and ctx statistics
accounting much more than it was on the first implementation.
Signed-off-by: NAndre Detsch <adetsch@br.ibm.com>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

Export per-context statistics in spufs.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

Provide load average information for spu context.  The format
is identical to /proc/loadavg, which is also where a lot of code
and concepts is borrowed from.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

Add a cpus_allowed allowed filed to struct spu_context so that we always
use the cpu mask of the owning thread instead of the one happening to
call into the scheduler.  Also use this information in
grab_runnable_context to avoid spurious wakeups.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

Update scheduling information on every spu_run to allow for setting
threads to realtime priority just before running them.  This requires
some slightly ugly code in spufs_run_spu because we can just update
the information unlocked if the spu is not runnable, but we need to
acquire the active_mutex when it is runnable to protect against
find_victim.  This locking scheme requires opencoding
spu_acquire_runnable in spufs_run_spu which actually is a nice cleanup
all by itself.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

Enable preemptive scheduling for non-RT contexts.

We use the same algorithms as the CPU scheduler to calculate the time
slice length, and for now we also use the same timeslice length as the
CPU scheduler. This might be not enough for good performance and can be
changed after some benchmarking.

Note that currently we do not boost the priority for contexts waiting
on the runqueue for a long time, so contexts with a higher nice value
could starve ones with less priority.  This could easily be fixed once
the rework of the spu lists that Luke and I discussed is done.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

Get rid of the scheduler workqueues that complicated things a lot to
a dedicated spu scheduler thread that gets woken by a traditional
scheduler tick.  By default this scheduler tick runs a HZ * 10, aka
one spu scheduler tick for every 10 cpu ticks.

Currently the tick is not disabled when we have less context than
available spus, but I will implement this later.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

Make sure the mapping_lock also protects access to the various address_space
pointers used for tearing down the ptes on a spu context switch.

Because unmap_mapping_range can sleep we need to turn mapping_lock from
a spinlock into a sleeping mutex.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>
Signed-off-by: NJeremy Kerr <jk@ozlabs.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

This adds an option to spufs when the kernel is configured for
4K page to give it the ability to use 64K pages for SPE local store
mappings.

Currently, we are optimistic and try order 4 allocations when creating
contexts. If that fails, the code will fallback to 4K automatically.
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: NPaul Mackerras <paulus@samba.org>

There is no reason for run_sema to be a struct semaphore.  Changing
it to a mutex and rename it accordingly.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

For quite a while now spu state is protected by a simple mutex instead
of the old rw_semaphore, and this means we can simplify the locking
around spu_setup_isolated a lot.

Instead of doing an spu_release before entering spu_setup_isolated and
then calling the complicated spu_acquire_exclusive we can now simply
enter the function locked an in guaranteed runnable state, so that the
only bit of spu_acquire_exclusive that's left is the call to
spu_unmap_mappings.

Similarly there's no more need to unlock and reacquire the state_mutex
when spu_setup_isolated is done, but we can always return with the
lock held and only drop it in spu_run_init in the failure case.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

A single context should only be woken once, and we should not have
more wakeups for a given priority than the number of contexts on
that runqueue position.

Also add some asserts to trap future problems in this area more
easily.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

Make sure the pointers to various mappings are cleared once the last
user stopped using them.  This avoids accessing freed memory when
tearing down the gang directory aswell as optimizing away
pte invalidations if no one uses these.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

For SCHED_RR tasks we can do some really trivial timeslicing.  Basically
we fire up a time for every scheduler tick that searches for a higher
or same priority thread that is on the runqueue and if there is one
context switches to it.  Because we can't lock spus from timer context
we actually run this from a delayed runqueue instead of a timer.

A nice optimization would be to skip the actual priority bitmap search
when there are less contexts than physical spus available.  To implement
this I need a so far unpublished patch from Andre, and it will be added
after we have that patch in.

Note that right now we only do the time slicing for SCHED_RR tasks.
The code would work for SCHED_OTHER tasks aswell, but their prio
value is defered from the one the PPU thread has at time of spu_run,
and using this for spu scheduling decisions would make the code very
unfair.  SCHED_OTHER support will be enabled once we the spu scheduler
knows how to calculcate cpu_context.prio (very soon)
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

If we start a spu context with realtime priority we want it to run
immediately and not wait until some other lower priority thread has
finished.  Try to find a suitable victim and use it's spu in this
case.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

There is no need to directly wake up contexts in spu_activate when
called from spu_run, so add a flag to surpress this wakeup.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

It doesn't make any sense to have a priority field in the physical spu
structure.  Move it into the spu context instead.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

Various cleanups in code surrounding the state semaphore:

 - inline spu_acquire/spu_release
 - cleanup spu_acquire_* and add kerneldoc comments to these functions
 - remove spu_release_exclusive and replace it with spu_release
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

The r/w semaphore to lock the spus was overkill and can be replaced
with a mutex to make it faster, simpler and easier to debug.  It also
helps to allow making most spufs interruptible in future patches.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

Only bind_context/unbind_context change the spu context state.  Thus
we can move all assignents of SPU_STATE_RUNNABLE into bind_context,
which parallels the unbind side aswell.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>

unbind_context already sets the context state to SPU_STATE_SAVED, thus
the spu_deactivate callers don't need to do it again.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NArnd Bergmann <arnd.bergmann@de.ibm.com>