1. 17 7月, 2010 1 次提交
  2. 22 6月, 2010 1 次提交
  3. 18 6月, 2010 1 次提交
  4. 09 6月, 2010 9 次提交
    • V
      sched: Change nohz idle load balancing logic to push model · 83cd4fe2
      Venkatesh Pallipadi 提交于
      In the new push model, all idle CPUs indeed go into nohz mode. There is
      still the concept of idle load balancer (performing the load balancing
      on behalf of all the idle cpu's in the system). Busy CPU kicks the nohz
      balancer when any of the nohz CPUs need idle load balancing.
      The kickee CPU does the idle load balancing on behalf of all idle CPUs
      instead of the normal idle balance.
      
      This addresses the below two problems with the current nohz ilb logic:
      * the idle load balancer continued to have periodic ticks during idle and
        wokeup frequently, even though it did not have any rebalancing to do on
        behalf of any of the idle CPUs.
      * On x86 and CPUs that have APIC timer stoppage on idle CPUs, this
        periodic wakeup can result in a periodic additional interrupt on a CPU
        doing the timer broadcast.
      
      Also currently we are migrating the unpinned timers from an idle to the cpu
      doing idle load balancing (when all the cpus in the system are idle,
      there is no idle load balancing cpu and timers get added to the same idle cpu
      where the request was made. So the existing optimization works only on semi idle
      system).
      
      And In semi idle system, we no longer have periodic ticks on the idle load
      balancer CPU. Using that cpu will add more delays to the timers than intended
      (as that cpu's timer base may not be uptodate wrt jiffies etc). This was
      causing mysterious slowdowns during boot etc.
      
      For now, in the semi idle case, use the nearest busy cpu for migrating timers
      from an idle cpu.  This is good for power-savings anyway.
      Signed-off-by: NVenkatesh Pallipadi <venki@google.com>
      Signed-off-by: NSuresh Siddha <suresh.b.siddha@intel.com>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      LKML-Reference: <1274486981.2840.46.camel@sbs-t61.sc.intel.com>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      83cd4fe2
    • V
      sched: Avoid side-effect of tickless idle on update_cpu_load · fdf3e95d
      Venkatesh Pallipadi 提交于
      tickless idle has a negative side effect on update_cpu_load(), which
      in turn can affect load balancing behavior.
      
      update_cpu_load() is supposed to be called every tick, to keep track
      of various load indicies. With tickless idle, there are no scheduler
      ticks called on the idle CPUs. Idle CPUs may still do load balancing
      (with idle_load_balance CPU) using the stale cpu_load. It will also
      cause problems when all CPUs go idle for a while and become active
      again. In this case loads would not degrade as expected.
      
      This is how rq->nr_load_updates change looks like under different
      conditions:
      
      <cpu_num> <nr_load_updates change>
      All CPUS idle for 10 seconds (HZ=1000)
      0 1621
      10 496
      11 139
      12 875
      13 1672
      14 12
      15 21
      1 1472
      2 2426
      3 1161
      4 2108
      5 1525
      6 701
      7 249
      8 766
      9 1967
      
      One CPU busy rest idle for 10 seconds
      0 10003
      10 601
      11 95
      12 966
      13 1597
      14 114
      15 98
      1 3457
      2 93
      3 6679
      4 1425
      5 1479
      6 595
      7 193
      8 633
      9 1687
      
      All CPUs busy for 10 seconds
      0 10026
      10 10026
      11 10026
      12 10026
      13 10025
      14 10025
      15 10025
      1 10026
      2 10026
      3 10026
      4 10026
      5 10026
      6 10026
      7 10026
      8 10026
      9 10026
      
      That is update_cpu_load works properly only when all CPUs are busy.
      If all are idle, all the CPUs get way lower updates.  And when few
      CPUs are busy and rest are idle, only busy and ilb CPU does proper
      updates and rest of the idle CPUs will do lower updates.
      
      The patch keeps track of when a last update was done and fixes up
      the load avg based on current time.
      
      On one of my test system SPECjbb with warehouse 1..numcpus, patch
      improves throughput numbers by ~1% (average of 6 runs).  On another
      test system (with different domain hierarchy) there is no noticable
      change in perf.
      Signed-off-by: NVenkatesh Pallipadi <venki@google.com>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      LKML-Reference: <AANLkTilLtDWQsAUrIxJ6s04WTgmw9GuOODc5AOrYsaR5@mail.gmail.com>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      fdf3e95d
    • O
      sched: Simplify the reacquire_kernel_lock() logic · 246d86b5
      Oleg Nesterov 提交于
      - Contrary to what 6d558c3a says, there is no need to reload
        prev = rq->curr after the context switch. You always schedule
        back to where you came from, prev must be equal to current
        even if cpu/rq was changed.
      
      - This also means reacquire_kernel_lock() can use prev instead
        of current.
      
      - No need to reassign switch_count if reacquire_kernel_lock()
        reports need_resched(), we can just move the initial assignment
        down, under the "need_resched_nonpreemptible:" label.
      
      - Try to update the comment after context_switch().
      Signed-off-by: NOleg Nesterov <oleg@redhat.com>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <20100519125711.GA30199@redhat.com>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      246d86b5
    • P
      sched_clock: Add local_clock() API and improve documentation · c676329a
      Peter Zijlstra 提交于
      For people who otherwise get to write: cpu_clock(smp_processor_id()),
      there is now: local_clock().
      
      Also, as per suggestion from Andrew, provide some documentation on
      the various clock interfaces, and minimize the unsigned long long vs
      u64 mess.
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Jens Axboe <jaxboe@fusionio.com>
      LKML-Reference: <1275052414.1645.52.camel@laptop>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      c676329a
    • T
      sched: add hooks for workqueue · 21aa9af0
      Tejun Heo 提交于
      Concurrency managed workqueue needs to know when workers are going to
      sleep and waking up.  Using these two hooks, cmwq keeps track of the
      current concurrency level and throttles execution of new works if it's
      too high and wakes up another worker from the sleep hook if it becomes
      too low.
      
      This patch introduces PF_WQ_WORKER to identify workqueue workers and
      adds the following two hooks.
      
      * wq_worker_waking_up(): called when a worker is woken up.
      
      * wq_worker_sleeping(): called when a worker is going to sleep and may
        return a pointer to a local task which should be woken up.  The
        returned task is woken up using try_to_wake_up_local() which is
        simplified ttwu which is called under rq lock and can only wake up
        local tasks.
      
      Both hooks are currently defined as noop in kernel/workqueue_sched.h.
      Later cmwq implementation will replace them with proper
      implementation.
      
      These hooks are hard coded as they'll always be enabled.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NPeter Zijlstra <peterz@infradead.org>
      Cc: Mike Galbraith <efault@gmx.de>
      Cc: Ingo Molnar <mingo@elte.hu>
      21aa9af0
    • T
      sched: refactor try_to_wake_up() · 9ed3811a
      Tejun Heo 提交于
      Factor ttwu_activate() and ttwu_woken_up() out of try_to_wake_up().
      The factoring out doesn't affect try_to_wake_up() much
      code-generation-wise.  Depending on configuration options, it ends up
      generating the same object code as before or slightly different one
      due to different register assignment.
      
      This is to help future implementation of try_to_wake_up_local().
      
      Mike Galbraith suggested rename to ttwu_post_activation() from
      ttwu_woken_up() and comment update in try_to_wake_up().
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NPeter Zijlstra <peterz@infradead.org>
      Cc: Mike Galbraith <efault@gmx.de>
      Cc: Ingo Molnar <mingo@elte.hu>
      9ed3811a
    • T
      sched: adjust when cpu_active and cpuset configurations are updated during cpu on/offlining · 3a101d05
      Tejun Heo 提交于
      Currently, when a cpu goes down, cpu_active is cleared before
      CPU_DOWN_PREPARE starts and cpuset configuration is updated from a
      default priority cpu notifier.  When a cpu is coming up, it's set
      before CPU_ONLINE but cpuset configuration again is updated from the
      same cpu notifier.
      
      For cpu notifiers, this presents an inconsistent state.  Threads which
      a CPU_DOWN_PREPARE notifier expects to be bound to the CPU can be
      migrated to other cpus because the cpu is no more inactive.
      
      Fix it by updating cpu_active in the highest priority cpu notifier and
      cpuset configuration in the second highest when a cpu is coming up.
      Down path is updated similarly.  This guarantees that all other cpu
      notifiers see consistent cpu_active and cpuset configuration.
      
      cpuset_track_online_cpus() notifier is converted to
      cpuset_update_active_cpus() which just updates the configuration and
      now called from cpuset_cpu_[in]active() notifiers registered from
      sched_init_smp().  If cpuset is disabled, cpuset_update_active_cpus()
      degenerates into partition_sched_domains() making separate notifier
      for !CONFIG_CPUSETS unnecessary.
      
      This problem is triggered by cmwq.  During CPU_DOWN_PREPARE, hotplug
      callback creates a kthread and kthread_bind()s it to the target cpu,
      and the thread is expected to run on that cpu.
      
      * Ingo's test discovered __cpuinit/exit markups were incorrect.
        Fixed.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Rusty Russell <rusty@rustcorp.com.au>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Paul Menage <menage@google.com>
      3a101d05
    • T
      sched: define and use CPU_PRI_* enums for cpu notifier priorities · 50a323b7
      Tejun Heo 提交于
      Instead of hardcoding priority 10 and 20 in sched and perf, collect
      them into CPU_PRI_* enums.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Rusty Russell <rusty@rustcorp.com.au>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
      50a323b7
    • P
      sched: Fix PROVE_RCU vs cpu_cgroup · dc61b1d6
      Peter Zijlstra 提交于
      PROVE_RCU has a few issues with the cpu_cgroup because the scheduler
      typically holds rq->lock around the css rcu derefs but the generic
      cgroup code doesn't (and can't) know about that lock.
      
      Provide means to add extra checks to the css dereference and use that
      in the scheduler to annotate its users.
      
      The addition of rq->lock to these checks is correct because the
      cgroup_subsys::attach() method takes the rq->lock for each task it
      moves, therefore by holding that lock, we ensure the task is pinned to
      the current cgroup and the RCU derefence is valid.
      
      That leaves one genuine race in __sched_setscheduler() where we used
      task_group() without holding any of the required locks and thus raced
      with the cgroup code. Solve this by moving the check under the
      appropriate lock.
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
      LKML-Reference: <new-submission>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      dc61b1d6
  5. 01 6月, 2010 1 次提交
  6. 30 5月, 2010 1 次提交
  7. 28 5月, 2010 1 次提交
  8. 21 5月, 2010 2 次提交
    • J
      kdb: core for kgdb back end (2 of 2) · 67fc4e0c
      Jason Wessel 提交于
      This patch contains the hooks and instrumentation into kernel which
      live outside the kernel/debug directory, which the kdb core
      will call to run commands like lsmod, dmesg, bt etc...
      
      CC: linux-arch@vger.kernel.org
      Signed-off-by: NJason Wessel <jason.wessel@windriver.com>
      Signed-off-by: NMartin Hicks <mort@sgi.com>
      67fc4e0c
    • M
      wait_event_interruptible_locked() interface · 22c43c81
      Michal Nazarewicz 提交于
      New wait_event_interruptible{,_exclusive}_locked{,_irq} macros added.
      They work just like versions without _locked* suffix but require the
      wait queue's lock to be held.  Also __wake_up_locked() is now exported
      as to pair it with the above macros.
      
      The use case of this new facility is when one uses wait queue's lock
      to  protect a data structure.  This may be advantageous if the
      structure needs to be protected by a spinlock anyway.  In particular,
      with additional spinlock the following code has to be used to wait
      for a condition:
      
      spin_lock(&data.lock);
      ...
      for (ret = 0; !ret && !(condition); ) {
      	spin_unlock(&data.lock);
      	ret = wait_event_interruptible(data.wqh, (condition));
      	spin_lock(&data.lock);
      }
      ...
      spin_unlock(&data.lock);
      
      This looks bizarre plus wait_event_interruptible() locks the wait
      queue's lock anyway so there is a unlock+lock sequence where it could
      be avoided.
      
      To avoid those problems and benefit from wait queue's lock, a code
      similar to the following should be used:
      
      /* Waiting */
      spin_lock(&data.wqh.lock);
      ...
      ret = wait_event_interruptible_locked(data.wqh, (condition));
      ...
      spin_unlock(&data.wqh.lock);
      
      /* Waiting exclusively */
      spin_lock(&data.whq.lock);
      ...
      ret = wait_event_interruptible_exclusive_locked(data.whq, (condition));
      ...
      spin_unlock(&data.whq.lock);
      
      /* Waking up */
      spin_lock(&data.wqh.lock);
      ...
      wake_up_locked(&data.wqh);
      ...
      spin_unlock(&data.wqh.lock);
      
      When spin_lock_irq() is used matching versions of macros need to be
      used (*_locked_irq()).
      Signed-off-by: NMichal Nazarewicz <m.nazarewicz@samsung.com>
      Cc: Kyungmin Park <kyungmin.park@samsung.com>
      Cc: Marek Szyprowski <m.szyprowski@samsung.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Takashi Iwai <tiwai@suse.de>
      Cc: David Howells <dhowells@redhat.com>
      Cc: Andreas Herrmann <andreas.herrmann3@amd.com>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Mike Galbraith <efault@gmx.de>
      Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>
      22c43c81
  9. 11 5月, 2010 2 次提交
    • C
      sched, wait: Use wrapper functions · a93d2f17
      Changli Gao 提交于
      epoll should not touch flags in wait_queue_t. This patch introduces a new
      function __add_wait_queue_exclusive(), for the users, who use wait queue as a
      LIFO queue.
      
      __add_wait_queue_tail_exclusive() is introduced too instead of
      add_wait_queue_exclusive_locked(). remove_wait_queue_locked() is removed, as
      it is a duplicate of __remove_wait_queue(), disliked by users, and with less
      users.
      Signed-off-by: NChangli Gao <xiaosuo@gmail.com>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Alexander Viro <viro@zeniv.linux.org.uk>
      Cc: Paul Menage <menage@google.com>
      Cc: Li Zefan <lizf@cn.fujitsu.com>
      Cc: Davide Libenzi <davidel@xmailserver.org>
      Cc: <containers@lists.linux-foundation.org>
      LKML-Reference: <1273214006-2979-1-git-send-email-xiaosuo@gmail.com>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      a93d2f17
    • P
      rcu: refactor RCU's context-switch handling · 25502a6c
      Paul E. McKenney 提交于
      The addition of preemptible RCU to treercu resulted in a bit of
      confusion and inefficiency surrounding the handling of context switches
      for RCU-sched and for RCU-preempt.  For RCU-sched, a context switch
      is a quiescent state, pure and simple, just like it always has been.
      For RCU-preempt, a context switch is in no way a quiescent state, but
      special handling is required when a task blocks in an RCU read-side
      critical section.
      
      However, the callout from the scheduler and the outer loop in ksoftirqd
      still calls something named rcu_sched_qs(), whose name is no longer
      accurate.  Furthermore, when rcu_check_callbacks() notes an RCU-sched
      quiescent state, it ends up unnecessarily (though harmlessly, aside
      from the performance hit) enqueuing the current task if it happens to
      be running in an RCU-preempt read-side critical section.  This not only
      increases the maximum latency of scheduler_tick(), it also needlessly
      increases the overhead of the next outermost rcu_read_unlock() invocation.
      
      This patch addresses this situation by separating the notion of RCU's
      context-switch handling from that of RCU-sched's quiescent states.
      The context-switch handling is covered by rcu_note_context_switch() in
      general and by rcu_preempt_note_context_switch() for preemptible RCU.
      This permits rcu_sched_qs() to handle quiescent states and only quiescent
      states.  It also reduces the maximum latency of scheduler_tick(), though
      probably by much less than a microsecond.  Finally, it means that tasks
      within preemptible-RCU read-side critical sections avoid incurring the
      overhead of queuing unless there really is a context switch.
      Suggested-by: NLai Jiangshan <laijs@cn.fujitsu.com>
      Acked-by: NLai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: NPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Peter Zijlstra <peterz@infradead.org>
      25502a6c
  10. 07 5月, 2010 5 次提交
    • P
      sched: Remove rq argument to the tracepoints · 27a9da65
      Peter Zijlstra 提交于
      struct rq isn't visible outside of sched.o so its near useless to
      expose the pointer, also there are no users of it, so remove it.
      Acked-by: NSteven Rostedt <rostedt@goodmis.org>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <1272997616.1642.207.camel@laptop>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      27a9da65
    • P
      rcu: need barrier() in UP synchronize_sched_expedited() · fc390cde
      Paul E. McKenney 提交于
      If synchronize_sched_expedited() is ever to be called from within
      kernel/sched.c in a !SMP PREEMPT kernel, the !SMP implementation needs
      a barrier().
      Signed-off-by: NPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Signed-off-by: NTejun Heo <tj@kernel.org>
      fc390cde
    • P
      sched: correctly place paranioa memory barriers in synchronize_sched_expedited() · cc631fb7
      Paul E. McKenney 提交于
      The memory barriers must be in the SMP case, not in the !SMP case.
      Also add a barrier after the atomic_inc() in order to ensure that
      other CPUs see post-synchronize_sched_expedited() actions as following
      the expedited grace period.
      Signed-off-by: NPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Signed-off-by: NTejun Heo <tj@kernel.org>
      cc631fb7
    • T
      sched: kill paranoia check in synchronize_sched_expedited() · 94458d5e
      Tejun Heo 提交于
      The paranoid check which verifies that the cpu_stop callback is
      actually called on all online cpus is completely superflous.  It's
      guaranteed by cpu_stop facility and if it didn't work as advertised
      other things would go horribly wrong and trying to recover using
      synchronize_sched() wouldn't be very meaningful.
      
      Kill the paranoid check.  Removal of this feature is done as a
      separate step so that it can serve as a bisection point if something
      actually goes wrong.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NPeter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Dipankar Sarma <dipankar@in.ibm.com>
      Cc: Josh Triplett <josh@freedesktop.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Dimitri Sivanich <sivanich@sgi.com>
      94458d5e
    • T
      sched: replace migration_thread with cpu_stop · 969c7921
      Tejun Heo 提交于
      Currently migration_thread is serving three purposes - migration
      pusher, context to execute active_load_balance() and forced context
      switcher for expedited RCU synchronize_sched.  All three roles are
      hardcoded into migration_thread() and determining which job is
      scheduled is slightly messy.
      
      This patch kills migration_thread and replaces all three uses with
      cpu_stop.  The three different roles of migration_thread() are
      splitted into three separate cpu_stop callbacks -
      migration_cpu_stop(), active_load_balance_cpu_stop() and
      synchronize_sched_expedited_cpu_stop() - and each use case now simply
      asks cpu_stop to execute the callback as necessary.
      
      synchronize_sched_expedited() was implemented with private
      preallocated resources and custom multi-cpu queueing and waiting
      logic, both of which are provided by cpu_stop.
      synchronize_sched_expedited_count is made atomic and all other shared
      resources along with the mutex are dropped.
      
      synchronize_sched_expedited() also implemented a check to detect cases
      where not all the callback got executed on their assigned cpus and
      fall back to synchronize_sched().  If called with cpu hotplug blocked,
      cpu_stop already guarantees that and the condition cannot happen;
      otherwise, stop_machine() would break.  However, this patch preserves
      the paranoid check using a cpumask to record on which cpus the stopper
      ran so that it can serve as a bisection point if something actually
      goes wrong theree.
      
      Because the internal execution state is no longer visible,
      rcu_expedited_torture_stats() is removed.
      
      This patch also renames cpu_stop threads to from "stopper/%d" to
      "migration/%d".  The names of these threads ultimately don't matter
      and there's no reason to make unnecessary userland visible changes.
      
      With this patch applied, stop_machine() and sched now share the same
      resources.  stop_machine() is faster without wasting any resources and
      sched migration users are much cleaner.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NPeter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Dipankar Sarma <dipankar@in.ibm.com>
      Cc: Josh Triplett <josh@freedesktop.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Dimitri Sivanich <sivanich@sgi.com>
      969c7921
  11. 30 4月, 2010 1 次提交
  12. 23 4月, 2010 3 次提交
  13. 15 4月, 2010 1 次提交
  14. 06 4月, 2010 1 次提交
    • A
      sched: Fix sched_getaffinity() · 84fba5ec
      Anton Blanchard 提交于
      taskset on 2.6.34-rc3 fails on one of my ppc64 test boxes with
      the following error:
      
        sched_getaffinity(0, 16, 0x10029650030) = -1 EINVAL (Invalid argument)
      
      This box has 128 threads and 16 bytes is enough to cover it.
      
      Commit cd3d8031 (sched:
      sched_getaffinity(): Allow less than NR_CPUS length) is
      comparing this 16 bytes agains nr_cpu_ids.
      
      Fix it by comparing nr_cpu_ids to the number of bits in the
      cpumask we pass in.
      Signed-off-by: NAnton Blanchard <anton@samba.org>
      Reviewed-by: NKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Sharyathi Nagesh <sharyath@in.ibm.com>
      Cc: Ulrich Drepper <drepper@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Jack Steiner <steiner@sgi.com>
      Cc: Russ Anderson <rja@sgi.com>
      Cc: Mike Travis <travis@sgi.com>
      LKML-Reference: <20100406070218.GM5594@kryten>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      84fba5ec
  15. 03 4月, 2010 10 次提交
    • P
      sched: Add enqueue/dequeue flags · 371fd7e7
      Peter Zijlstra 提交于
      In order to reduce the dependency on TASK_WAKING rework the enqueue
      interface to support a proper flags field.
      
      Replace the int wakeup, bool head arguments with an int flags argument
      and create the following flags:
      
        ENQUEUE_WAKEUP - the enqueue is a wakeup of a sleeping task,
        ENQUEUE_WAKING - the enqueue has relative vruntime due to
                         having sched_class::task_waking() called,
        ENQUEUE_HEAD - the waking task should be places on the head
                       of the priority queue (where appropriate).
      
      For symmetry also convert sched_class::dequeue() to a flags scheme.
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <new-submission>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      371fd7e7
    • P
      sched: Fix nr_uninterruptible count · cc87f76a
      Peter Zijlstra 提交于
      The cpuload calculation in calc_load_account_active() assumes
      rq->nr_uninterruptible will not change on an offline cpu after
      migrate_nr_uninterruptible(). However the recent migrate on wakeup
      changes broke that and would result in decrementing the offline cpu's
      rq->nr_uninterruptible.
      
      Fix this by accounting the nr_uninterruptible on the waking cpu.
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <new-submission>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      cc87f76a
    • P
      sched: Optimize task_rq_lock() · 65cc8e48
      Peter Zijlstra 提交于
      Now that we hold the rq->lock over set_task_cpu() again, we can do
      away with most of the TASK_WAKING checks and reduce them again to
      set_cpus_allowed_ptr().
      
      Removes some conditionals from scheduling hot-paths.
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Oleg Nesterov <oleg@redhat.com>
      LKML-Reference: <new-submission>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      65cc8e48
    • P
      sched: Fix TASK_WAKING vs fork deadlock · 0017d735
      Peter Zijlstra 提交于
      Oleg noticed a few races with the TASK_WAKING usage on fork.
      
       - since TASK_WAKING is basically a spinlock, it should be IRQ safe
       - since we set TASK_WAKING (*) without holding rq->lock it could
         be there still is a rq->lock holder, thereby not actually
         providing full serialization.
      
      (*) in fact we clear PF_STARTING, which in effect enables TASK_WAKING.
      
      Cure the second issue by not setting TASK_WAKING in sched_fork(), but
      only temporarily in wake_up_new_task() while calling select_task_rq().
      
      Cure the first by holding rq->lock around the select_task_rq() call,
      this will disable IRQs, this however requires that we push down the
      rq->lock release into select_task_rq_fair()'s cgroup stuff.
      
      Because select_task_rq_fair() still needs to drop the rq->lock we
      cannot fully get rid of TASK_WAKING.
      Reported-by: NOleg Nesterov <oleg@redhat.com>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <new-submission>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      0017d735
    • O
      sched: Make select_fallback_rq() cpuset friendly · 9084bb82
      Oleg Nesterov 提交于
      Introduce cpuset_cpus_allowed_fallback() helper to fix the cpuset problems
      with select_fallback_rq(). It can be called from any context and can't use
      any cpuset locks including task_lock(). It is called when the task doesn't
      have online cpus in ->cpus_allowed but ttwu/etc must be able to find a
      suitable cpu.
      
      I am not proud of this patch. Everything which needs such a fat comment
      can't be good even if correct. But I'd prefer to not change the locking
      rules in the code I hardly understand, and in any case I believe this
      simple change make the code much more correct compared to deadlocks we
      currently have.
      Signed-off-by: NOleg Nesterov <oleg@redhat.com>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <20100315091027.GA9155@redhat.com>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      9084bb82
    • O
      sched: _cpu_down(): Don't play with current->cpus_allowed · 6a1bdc1b
      Oleg Nesterov 提交于
      _cpu_down() changes the current task's affinity and then recovers it at
      the end. The problems are well known: we can't restore old_allowed if it
      was bound to the now-dead-cpu, and we can race with the userspace which
      can change cpu-affinity during unplug.
      
      _cpu_down() should not play with current->cpus_allowed at all. Instead,
      take_cpu_down() can migrate the caller of _cpu_down() after __cpu_disable()
      removes the dying cpu from cpu_online_mask.
      Signed-off-by: NOleg Nesterov <oleg@redhat.com>
      Acked-by: NRafael J. Wysocki <rjw@sisk.pl>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <20100315091023.GA9148@redhat.com>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      6a1bdc1b
    • O
      sched: sched_exec(): Remove the select_fallback_rq() logic · 30da688e
      Oleg Nesterov 提交于
      sched_exec()->select_task_rq() reads/updates ->cpus_allowed lockless.
      This can race with other CPUs updating our ->cpus_allowed, and this
      looks meaningless to me.
      
      The task is current and running, it must have online cpus in ->cpus_allowed,
      the fallback mode is bogus. And, if ->sched_class returns the "wrong" cpu,
      this likely means we raced with set_cpus_allowed() which was called
      for reason, why should sched_exec() retry and call ->select_task_rq()
      again?
      
      Change the code to call sched_class->select_task_rq() directly and do
      nothing if the returned cpu is wrong after re-checking under rq->lock.
      
      From now task_struct->cpus_allowed is always stable under TASK_WAKING,
      select_fallback_rq() is always called under rq-lock or the caller or
      the caller owns TASK_WAKING (select_task_rq).
      Signed-off-by: NOleg Nesterov <oleg@redhat.com>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <20100315091019.GA9141@redhat.com>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      30da688e
    • O
      sched: move_task_off_dead_cpu(): Remove retry logic · c1804d54
      Oleg Nesterov 提交于
      The previous patch preserved the retry logic, but it looks unneeded.
      
      __migrate_task() can only fail if we raced with migration after we dropped
      the lock, but in this case the caller of set_cpus_allowed/etc must initiate
      migration itself if ->on_rq == T.
      
      We already fixed p->cpus_allowed, the changes in active/online masks must
      be visible to racer, it should migrate the task to online cpu correctly.
      Signed-off-by: NOleg Nesterov <oleg@redhat.com>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <20100315091014.GA9138@redhat.com>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      c1804d54
    • O
      sched: move_task_off_dead_cpu(): Take rq->lock around select_fallback_rq() · 1445c08d
      Oleg Nesterov 提交于
      move_task_off_dead_cpu()->select_fallback_rq() reads/updates ->cpus_allowed
      lockless. We can race with set_cpus_allowed() running in parallel.
      
      Change it to take rq->lock around select_fallback_rq(). Note that it is not
      trivial to move this spin_lock() into select_fallback_rq(), we must recheck
      the task was not migrated after we take the lock and other callers do not
      need this lock.
      
      To avoid the races with other callers of select_fallback_rq() which rely on
      TASK_WAKING, we also check p->state != TASK_WAKING and do nothing otherwise.
      The owner of TASK_WAKING must update ->cpus_allowed and choose the correct
      CPU anyway, and the subsequent __migrate_task() is just meaningless because
      p->se.on_rq must be false.
      
      Alternatively, we could change select_task_rq() to take rq->lock right
      after it calls sched_class->select_task_rq(), but this looks a bit ugly.
      
      Also, change it to not assume irqs are disabled and absorb __migrate_task_irq().
      Signed-off-by: NOleg Nesterov <oleg@redhat.com>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <20100315091010.GA9131@redhat.com>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      1445c08d
    • O
      sched: Kill the broken and deadlockable cpuset_lock/cpuset_cpus_allowed_locked code · 897f0b3c
      Oleg Nesterov 提交于
      This patch just states the fact the cpusets/cpuhotplug interaction is
      broken and removes the deadlockable code which only pretends to work.
      
      - cpuset_lock() doesn't really work. It is needed for
        cpuset_cpus_allowed_locked() but we can't take this lock in
        try_to_wake_up()->select_fallback_rq() path.
      
      - cpuset_lock() is deadlockable. Suppose that a task T bound to CPU takes
        callback_mutex. If cpu_down(CPU) happens before T drops callback_mutex
        stop_machine() preempts T, then migration_call(CPU_DEAD) tries to take
        cpuset_lock() and hangs forever because CPU is already dead and thus
        T can't be scheduled.
      
      - cpuset_cpus_allowed_locked() is deadlockable too. It takes task_lock()
        which is not irq-safe, but try_to_wake_up() can be called from irq.
      
      Kill them, and change select_fallback_rq() to use cpu_possible_mask, like
      we currently do without CONFIG_CPUSETS.
      
      Also, with or without this patch, with or without CONFIG_CPUSETS, the
      callers of select_fallback_rq() can race with each other or with
      set_cpus_allowed() pathes.
      
      The subsequent patches try to to fix these problems.
      Signed-off-by: NOleg Nesterov <oleg@redhat.com>
      Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <20100315091003.GA9123@redhat.com>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      897f0b3c