Skip to content

K
Kernel

openeuler / Kernel
接近 2 年 前同步成功

通知 8
Star 0
Fork 0
K
Kernel
提交 faafcba3 编写于 作者: L Linus Torvalds
浏览文件
操作

Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 

Pull scheduler updates from Ingo Molnar:
 "The main changes in this cycle were:

   - Optimized support for Intel "Cluster-on-Die" (CoD) topologies (Dave
     Hansen)

   - Various sched/idle refinements for better idle handling (Nicolas
     Pitre, Daniel Lezcano, Chuansheng Liu, Vincent Guittot)

   - sched/numa updates and optimizations (Rik van Riel)

   - sysbench speedup (Vincent Guittot)

   - capacity calculation cleanups/refactoring (Vincent Guittot)

   - Various cleanups to thread group iteration (Oleg Nesterov)

   - Double-rq-lock removal optimization and various refactorings
     (Kirill Tkhai)

   - various sched/deadline fixes

  ... and lots of other changes"

* 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (72 commits)
  sched/dl: Use dl_bw_of() under rcu_read_lock_sched()
  sched/fair: Delete resched_cpu() from idle_balance()
  sched, time: Fix build error with 64 bit cputime_t on 32 bit systems
  sched: Improve sysbench performance by fixing spurious active migration
  sched/x86: Fix up typo in topology detection
  x86, sched: Add new topology for multi-NUMA-node CPUs
  sched/rt: Use resched_curr() in task_tick_rt()
  sched: Use rq->rd in sched_setaffinity() under RCU read lock
  sched: cleanup: Rename 'out_unlock' to 'out_free_new_mask'
  sched: Use dl_bw_of() under RCU read lock
  sched/fair: Remove duplicate code from can_migrate_task()
  sched, mips, ia64: Remove __ARCH_WANT_UNLOCKED_CTXSW
  sched: print_rq(): Don't use tasklist_lock
  sched: normalize_rt_tasks(): Don't use _irqsave for tasklist_lock, use task_rq_lock()
  sched: Fix the task-group check in tg_has_rt_tasks()
  sched/fair: Leverage the idle state info when choosing the "idlest" cpu
  sched: Let the scheduler see CPU idle states
  sched/deadline: Fix inter- exclusive cpusets migrations
  sched/deadline: Clear dl_entity params when setscheduling to different class
  sched/numa: Kill the wrong/dead TASK_DEAD check in task_numa_fault()
  ...
上级 13ead805 f10e00f4
展开全部 隐藏空白更改
内联 并排
Showing with 1075 addition and 552 deletion
Documentation/scheduler/sched-deadline.txt
浏览文件 @ faafcba3
......@@ -15,6 +15,8 @@ CONTENTS
5. Tasks CPU affinity
5.1 SCHED_DEADLINE and cpusets HOWTO
6. Future plans
A. Test suite
B. Minimal main()
0. WARNING
......@@ -38,24 +40,25 @@ CONTENTS
==================
SCHED_DEADLINE uses three parameters, named "runtime", "period", and
"deadline" to schedule tasks. A SCHED_DEADLINE task is guaranteed to receive
"deadline", to schedule tasks. A SCHED_DEADLINE task should receive
"runtime" microseconds of execution time every "period" microseconds, and
these "runtime" microseconds are available within "deadline" microseconds
from the beginning of the period. In order to implement this behaviour,
every time the task wakes up, the scheduler computes a "scheduling deadline"
consistent with the guarantee (using the CBS[2,3] algorithm). Tasks are then
scheduled using EDF[1] on these scheduling deadlines (the task with the
smallest scheduling deadline is selected for execution). Notice that this
guaranteed is respected if a proper "admission control" strategy (see Section
"4. Bandwidth management") is used.
earliest scheduling deadline is selected for execution). Notice that the
task actually receives "runtime" time units within "deadline" if a proper
"admission control" strategy (see Section "4. Bandwidth management") is used
(clearly, if the system is overloaded this guarantee cannot be respected).
Summing up, the CBS[2,3] algorithms assigns scheduling deadlines to tasks so
that each task runs for at most its runtime every period, avoiding any
interference between different tasks (bandwidth isolation), while the EDF[1]
algorithm selects the task with the smallest scheduling deadline as the one
to be executed first. Thanks to this feature, also tasks that do not
strictly comply with the "traditional" real-time task model (see Section 3)
can effectively use the new policy.
algorithm selects the task with the earliest scheduling deadline as the one
to be executed next. Thanks to this feature, tasks that do not strictly comply
with the "traditional" real-time task model (see Section 3) can effectively
use the new policy.
In more details, the CBS algorithm assigns scheduling deadlines to
tasks in the following way:
......@@ -64,45 +67,45 @@ CONTENTS
"deadline", and "period" parameters;
- The state of the task is described by a "scheduling deadline", and
a "current runtime". These two parameters are initially set to 0;
a "remaining runtime". These two parameters are initially set to 0;
- When a SCHED_DEADLINE task wakes up (becomes ready for execution),
the scheduler checks if
current runtime runtime
---------------------------------- > ----------------
scheduling deadline - current time period
remaining runtime runtime
---------------------------------- > ---------
scheduling deadline - current time period
then, if the scheduling deadline is smaller than the current time, or
this condition is verified, the scheduling deadline and the
current budget are re-initialised as
remaining runtime are re-initialised as
scheduling deadline = current time + deadline
current runtime = runtime
remaining runtime = runtime
otherwise, the scheduling deadline and the current runtime are
otherwise, the scheduling deadline and the remaining runtime are
left unchanged;
- When a SCHED_DEADLINE task executes for an amount of time t, its
current runtime is decreased as
remaining runtime is decreased as
current runtime = current runtime - t
remaining runtime = remaining runtime - t
(technically, the runtime is decreased at every tick, or when the
task is descheduled / preempted);
- When the current runtime becomes less or equal than 0, the task is
- When the remaining runtime becomes less or equal than 0, the task is
said to be "throttled" (also known as "depleted" in real-time literature)
and cannot be scheduled until its scheduling deadline. The "replenishment
time" for this task (see next item) is set to be equal to the current
value of the scheduling deadline;
- When the current time is equal to the replenishment time of a
throttled task, the scheduling deadline and the current runtime are
throttled task, the scheduling deadline and the remaining runtime are
updated as
scheduling deadline = scheduling deadline + period
current runtime = current runtime + runtime
remaining runtime = remaining runtime + runtime
3. Scheduling Real-Time Tasks
......@@ -134,6 +137,50 @@ CONTENTS
A real-time task can be periodic with period P if r_{j+1} = r_j + P, or
sporadic with minimum inter-arrival time P is r_{j+1} >= r_j + P. Finally,
d_j = r_j + D, where D is the task's relative deadline.
The utilisation of a real-time task is defined as the ratio between its
WCET and its period (or minimum inter-arrival time), and represents
the fraction of CPU time needed to execute the task.
If the total utilisation sum_i(WCET_i/P_i) is larger than M (with M equal
to the number of CPUs), then the scheduler is unable to respect all the
deadlines.
Note that total utilisation is defined as the sum of the utilisations
WCET_i/P_i over all the real-time tasks in the system. When considering
multiple real-time tasks, the parameters of the i-th task are indicated
with the "_i" suffix.
Moreover, if the total utilisation is larger than M, then we risk starving
non- real-time tasks by real-time tasks.
If, instead, the total utilisation is smaller than M, then non real-time
tasks will not be starved and the system might be able to respect all the
deadlines.
As a matter of fact, in this case it is possible to provide an upper bound
for tardiness (defined as the maximum between 0 and the difference
between the finishing time of a job and its absolute deadline).
More precisely, it can be proven that using a global EDF scheduler the
maximum tardiness of each task is smaller or equal than
((M − 1) · WCET_max − WCET_min)/(M − (M − 2) · U_max) + WCET_max
where WCET_max = max_i{WCET_i} is the maximum WCET, WCET_min=min_i{WCET_i}
is the minimum WCET, and U_max = max_i{WCET_i/P_i} is the maximum utilisation.
If M=1 (uniprocessor system), or in case of partitioned scheduling (each
real-time task is statically assigned to one and only one CPU), it is
possible to formally check if all the deadlines are respected.
If D_i = P_i for all tasks, then EDF is able to respect all the deadlines
of all the tasks executing on a CPU if and only if the total utilisation
of the tasks running on such a CPU is smaller or equal than 1.
If D_i != P_i for some task, then it is possible to define the density of
a task as C_i/min{D_i,T_i}, and EDF is able to respect all the deadlines
of all the tasks running on a CPU if the sum sum_i C_i/min{D_i,T_i} of the
densities of the tasks running on such a CPU is smaller or equal than 1
(notice that this condition is only sufficient, and not necessary).
On multiprocessor systems with global EDF scheduling (non partitioned
systems), a sufficient test for schedulability can not be based on the
utilisations (it can be shown that task sets with utilisations slightly
larger than 1 can miss deadlines regardless of the number of CPUs M).
However, as previously stated, enforcing that the total utilisation is smaller
than M is enough to guarantee that non real-time tasks are not starved and
that the tardiness of real-time tasks has an upper bound.
SCHED_DEADLINE can be used to schedule real-time tasks guaranteeing that
the jobs' deadlines of a task are respected. In order to do this, a task
......@@ -147,6 +194,8 @@ CONTENTS
and the absolute deadlines (d_j) coincide, so a proper admission control
allows to respect the jobs' absolute deadlines for this task (this is what is
called "hard schedulability property" and is an extension of Lemma 1 of [2]).
Notice that if runtime > deadline the admission control will surely reject
this task, as it is not possible to respect its temporal constraints.
References:
1 - C. L. Liu and J. W. Layland. Scheduling algorithms for multiprogram-
......@@ -156,46 +205,57 @@ CONTENTS
Real-Time Systems. Proceedings of the 19th IEEE Real-time Systems
Symposium, 1998. http://retis.sssup.it/~giorgio/paps/1998/rtss98-cbs.pdf
3 - L. Abeni. Server Mechanisms for Multimedia Applications. ReTiS Lab
Technical Report. http://xoomer.virgilio.it/lucabe72/pubs/tr-98-01.ps
Technical Report. http://disi.unitn.it/~abeni/tr-98-01.pdf
4. Bandwidth management
=======================
In order for the -deadline scheduling to be effective and useful, it is
important to have some method to keep the allocation of the available CPU
bandwidth to the tasks under control.
This is usually called "admission control" and if it is not performed at all,
As previously mentioned, in order for -deadline scheduling to be
effective and useful (that is, to be able to provide "runtime" time units
within "deadline"), it is important to have some method to keep the allocation
of the available fractions of CPU time to the various tasks under control.
This is usually called "admission control" and if it is not performed, then
no guarantee can be given on the actual scheduling of the -deadline tasks.
Since when RT-throttling has been introduced each task group has a bandwidth
associated, calculated as a certain amount of runtime over a period.
Moreover, to make it possible to manipulate such bandwidth, readable/writable
controls have been added to both procfs (for system wide settings) and cgroupfs
(for per-group settings).
Therefore, the same interface is being used for controlling the bandwidth
distrubution to -deadline tasks.
However, more discussion is needed in order to figure out how we want to manage
SCHED_DEADLINE bandwidth at the task group level. Therefore, SCHED_DEADLINE
uses (for now) a less sophisticated, but actually very sensible, mechanism to
ensure that a certain utilization cap is not overcome per each root_domain.
Another main difference between deadline bandwidth management and RT-throttling
As already stated in Section 3, a necessary condition to be respected to
correctly schedule a set of real-time tasks is that the total utilisation
is smaller than M. When talking about -deadline tasks, this requires that
the sum of the ratio between runtime and period for all tasks is smaller
than M. Notice that the ratio runtime/period is equivalent to the utilisation
of a "traditional" real-time task, and is also often referred to as
"bandwidth".
The interface used to control the CPU bandwidth that can be allocated
to -deadline tasks is similar to the one already used for -rt
tasks with real-time group scheduling (a.k.a. RT-throttling - see
Documentation/scheduler/sched-rt-group.txt), and is based on readable/
writable control files located in procfs (for system wide settings).
Notice that per-group settings (controlled through cgroupfs) are still not
defined for -deadline tasks, because more discussion is needed in order to
figure out how we want to manage SCHED_DEADLINE bandwidth at the task group
level.
A main difference between deadline bandwidth management and RT-throttling
is that -deadline tasks have bandwidth on their own (while -rt ones don't!),
and thus we don't need an higher level throttling mechanism to enforce the
desired bandwidth.
and thus we don't need a higher level throttling mechanism to enforce the
desired bandwidth. In other words, this means that interface parameters are
only used at admission control time (i.e., when the user calls
sched_setattr()). Scheduling is then performed considering actual tasks'
parameters, so that CPU bandwidth is allocated to SCHED_DEADLINE tasks
respecting their needs in terms of granularity. Therefore, using this simple
interface we can put a cap on total utilization of -deadline tasks (i.e.,
\Sum (runtime_i / period_i) < global_dl_utilization_cap).
4.1 System wide settings
------------------------
The system wide settings are configured under the /proc virtual file system.
For now the -rt knobs are used for dl admission control and the -deadline
runtime is accounted against the -rt runtime. We realise that this isn't
entirely desirable; however, it is better to have a small interface for now,
and be able to change it easily later. The ideal situation (see 5.) is to run
-rt tasks from a -deadline server; in which case the -rt bandwidth is a direct
subset of dl_bw.
For now the -rt knobs are used for -deadline admission control and the
-deadline runtime is accounted against the -rt runtime. We realise that this
isn't entirely desirable; however, it is better to have a small interface for
now, and be able to change it easily later. The ideal situation (see 5.) is to
run -rt tasks from a -deadline server; in which case the -rt bandwidth is a
direct subset of dl_bw.
This means that, for a root_domain comprising M CPUs, -deadline tasks
can be created while the sum of their bandwidths stays below:
......@@ -231,8 +291,16 @@ CONTENTS
950000. With rt_period equal to 1000000, by default, it means that -deadline
tasks can use at most 95%, multiplied by the number of CPUs that compose the
root_domain, for each root_domain.
This means that non -deadline tasks will receive at least 5% of the CPU time,
and that -deadline tasks will receive their runtime with a guaranteed
worst-case delay respect to the "deadline" parameter. If "deadline" = "period"
and the cpuset mechanism is used to implement partitioned scheduling (see
Section 5), then this simple setting of the bandwidth management is able to
deterministically guarantee that -deadline tasks will receive their runtime
in a period.
A -deadline task cannot fork.
Finally, notice that in order not to jeopardize the admission control a
-deadline task cannot fork.
5. Tasks CPU affinity
=====================
......@@ -279,3 +347,179 @@ CONTENTS
throttling patches [https://lkml.org/lkml/2010/2/23/239] but we still are in
the preliminary phases of the merge and we really seek feedback that would
help us decide on the direction it should take.
Appendix A. Test suite
======================
The SCHED_DEADLINE policy can be easily tested using two applications that
are part of a wider Linux Scheduler validation suite. The suite is
available as a GitHub repository: https://github.com/scheduler-tools.
The first testing application is called rt-app and can be used to
start multiple threads with specific parameters. rt-app supports
SCHED_{OTHER,FIFO,RR,DEADLINE} scheduling policies and their related
parameters (e.g., niceness, priority, runtime/deadline/period). rt-app
is a valuable tool, as it can be used to synthetically recreate certain
workloads (maybe mimicking real use-cases) and evaluate how the scheduler
behaves under such workloads. In this way, results are easily reproducible.
rt-app is available at: https://github.com/scheduler-tools/rt-app.
Thread parameters can be specified from the command line, with something like
this:
# rt-app -t 100000:10000:d -t 150000:20000:f:10 -D5
The above creates 2 threads. The first one, scheduled by SCHED_DEADLINE,
executes for 10ms every 100ms. The second one, scheduled at SCHED_FIFO
priority 10, executes for 20ms every 150ms. The test will run for a total
of 5 seconds.
More interestingly, configurations can be described with a json file that
can be passed as input to rt-app with something like this:
# rt-app my_config.json
The parameters that can be specified with the second method are a superset
of the command line options. Please refer to rt-app documentation for more
details (<rt-app-sources>/doc/*.json).
The second testing application is a modification of schedtool, called
schedtool-dl, which can be used to setup SCHED_DEADLINE parameters for a
certain pid/application. schedtool-dl is available at:
https://github.com/scheduler-tools/schedtool-dl.git.
The usage is straightforward:
# schedtool -E -t 10000000:100000000 -e ./my_cpuhog_app
With this, my_cpuhog_app is put to run inside a SCHED_DEADLINE reservation
of 10ms every 100ms (note that parameters are expressed in microseconds).
You can also use schedtool to create a reservation for an already running
application, given that you know its pid:
# schedtool -E -t 10000000:100000000 my_app_pid
Appendix B. Minimal main()
==========================
We provide in what follows a simple (ugly) self-contained code snippet
showing how SCHED_DEADLINE reservations can be created by a real-time
application developer.
#define _GNU_SOURCE
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <linux/unistd.h>
#include <linux/kernel.h>
#include <linux/types.h>
#include <sys/syscall.h>
#include <pthread.h>
#define gettid() syscall(__NR_gettid)
#define SCHED_DEADLINE 6
/* XXX use the proper syscall numbers */
#ifdef __x86_64__
#define __NR_sched_setattr 314
#define __NR_sched_getattr 315
#endif
#ifdef __i386__
#define __NR_sched_setattr 351
#define __NR_sched_getattr 352
#endif
#ifdef __arm__
#define __NR_sched_setattr 380
#define __NR_sched_getattr 381
#endif
static volatile int done;
struct sched_attr {
__u32 size;
__u32 sched_policy;
__u64 sched_flags;
/* SCHED_NORMAL, SCHED_BATCH */
__s32 sched_nice;
/* SCHED_FIFO, SCHED_RR */
__u32 sched_priority;
/* SCHED_DEADLINE (nsec) */
__u64 sched_runtime;
__u64 sched_deadline;
__u64 sched_period;
};
int sched_setattr(pid_t pid,
const struct sched_attr *attr,
unsigned int flags)
{
return syscall(__NR_sched_setattr, pid, attr, flags);
}
int sched_getattr(pid_t pid,
struct sched_attr *attr,
unsigned int size,
unsigned int flags)
{
return syscall(__NR_sched_getattr, pid, attr, size, flags);
}
void *run_deadline(void *data)
{
struct sched_attr attr;
int x = 0;
int ret;
unsigned int flags = 0;
printf("deadline thread started [%ld]\n", gettid());
attr.size = sizeof(attr);
attr.sched_flags = 0;
attr.sched_nice = 0;
attr.sched_priority = 0;
/* This creates a 10ms/30ms reservation */
attr.sched_policy = SCHED_DEADLINE;
attr.sched_runtime = 10 * 1000 * 1000;
attr.sched_period = attr.sched_deadline = 30 * 1000 * 1000;
ret = sched_setattr(0, &attr, flags);
if (ret < 0) {
done = 0;
perror("sched_setattr");
exit(-1);
}
while (!done) {
x++;
}
printf("deadline thread dies [%ld]\n", gettid());
return NULL;
}
int main (int argc, char **argv)
{
pthread_t thread;
printf("main thread [%ld]\n", gettid());
pthread_create(&thread, NULL, run_deadline, NULL);
sleep(10);
done = 1;
pthread_join(thread, NULL);
printf("main dies [%ld]\n", gettid());
return 0;
}
arch/arm/kernel/topology.c
浏览文件 @ faafcba3
......@@ -42,7 +42,7 @@
*/
static DEFINE_PER_CPU(unsigned long, cpu_scale);
unsigned long arch_scale_freq_capacity(struct sched_domain *sd, int cpu)
unsigned long arch_scale_cpu_capacity(struct sched_domain *sd, int cpu)
{
return per_cpu(cpu_scale, cpu);
}
......@@ -166,7 +166,7 @@ static void update_cpu_capacity(unsigned int cpu)
set_capacity_scale(cpu, cpu_capacity(cpu) / middle_capacity);
printk(KERN_INFO "CPU%u: update cpu_capacity %lu\n",
cpu, arch_scale_freq_capacity(NULL, cpu));
cpu, arch_scale_cpu_capacity(NULL, cpu));
}
#else
......
arch/cris/arch-v10/drivers/sync_serial.c
浏览文件 @ faafcba3
......@@ -1086,7 +1086,6 @@ static ssize_t sync_serial_write(struct file *file, const char *buf,
}
local_irq_restore(flags);
schedule();
set_current_state(TASK_RUNNING);
remove_wait_queue(&port->out_wait_q, &wait);
if (signal_pending(current))
return -EINTR;
......
arch/cris/arch-v32/drivers/sync_serial.c
浏览文件 @ faafcba3
......@@ -1089,7 +1089,6 @@ static ssize_t sync_serial_write(struct file *file, const char *buf,
}
schedule();
set_current_state(TASK_RUNNING);
remove_wait_queue(&port->out_wait_q, &wait);
if (signal_pending(current))
......
arch/ia64/include/asm/processor.h
浏览文件 @ faafcba3
......@@ -19,7 +19,6 @@
#include <asm/ptrace.h>
#include <asm/ustack.h>
#define __ARCH_WANT_UNLOCKED_CTXSW
#define ARCH_HAS_PREFETCH_SWITCH_STACK
#define IA64_NUM_PHYS_STACK_REG 96
......
arch/mips/include/asm/processor.h
浏览文件 @ faafcba3
......@@ -397,12 +397,6 @@ unsigned long get_wchan(struct task_struct *p);
#define ARCH_HAS_PREFETCHW
#define prefetchw(x) __builtin_prefetch((x), 1, 1)
/*
* See Documentation/scheduler/sched-arch.txt; prevents deadlock on SMP
* systems.
*/
#define __ARCH_WANT_UNLOCKED_CTXSW
#endif
#endif /* _ASM_PROCESSOR_H */
arch/powerpc/include/asm/cputime.h
浏览文件 @ faafcba3
......@@ -32,6 +32,8 @@ static inline void setup_cputime_one_jiffy(void) { }
typedef u64 __nocast cputime_t;
typedef u64 __nocast cputime64_t;
#define cmpxchg_cputime(ptr, old, new) cmpxchg(ptr, old, new)
#ifdef __KERNEL__
/*
......
arch/powerpc/mm/fault.c
浏览文件 @ faafcba3
......@@ -30,7 +30,6 @@
#include <linux/kprobes.h>
#include <linux/kdebug.h>
#include <linux/perf_event.h>
#include <linux/magic.h>
#include <linux/ratelimit.h>
#include <linux/context_tracking.h>
#include <linux/hugetlb.h>
......@@ -521,7 +520,6 @@ int __kprobes do_page_fault(struct pt_regs *regs, unsigned long address,
void bad_page_fault(struct pt_regs *regs, unsigned long address, int sig)
{
const struct exception_table_entry *entry;
unsigned long *stackend;
/* Are we prepared to handle this fault? */
if ((entry = search_exception_tables(regs->nip)) != NULL) {
......@@ -550,8 +548,7 @@ void bad_page_fault(struct pt_regs *regs, unsigned long address, int sig)
printk(KERN_ALERT "Faulting instruction address: 0x%08lx\n",
regs->nip);
stackend = end_of_stack(current);
if (current != &init_task && *stackend != STACK_END_MAGIC)
if (task_stack_end_corrupted(current))
printk(KERN_ALERT "Thread overran stack, or stack corrupted\n");
die("Kernel access of bad area", regs, sig);
......
arch/s390/include/asm/cputime.h
浏览文件 @ faafcba3
......@@ -18,6 +18,8 @@
typedef unsigned long long __nocast cputime_t;
typedef unsigned long long __nocast cputime64_t;
#define cmpxchg_cputime(ptr, old, new) cmpxchg64(ptr, old, new)
static inline unsigned long __div(unsigned long long n, unsigned long base)
{
#ifndef CONFIG_64BIT
......
arch/um/drivers/random.c
浏览文件 @ faafcba3
......@@ -79,7 +79,6 @@ static ssize_t rng_dev_read (struct file *filp, char __user *buf, size_t size,
set_task_state(current, TASK_INTERRUPTIBLE);
schedule();
set_task_state(current, TASK_RUNNING);
remove_wait_queue(&host_read_wait, &wait);
if (atomic_dec_and_test(&host_sleep_count)) {
......
arch/x86/kernel/smpboot.c
浏览文件 @ faafcba3
......@@ -295,12 +295,20 @@ void smp_store_cpu_info(int id)
identify_secondary_cpu(c);
}
static bool
topology_same_node(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o)
{
int cpu1 = c->cpu_index, cpu2 = o->cpu_index;
return (cpu_to_node(cpu1) == cpu_to_node(cpu2));
}
static bool
topology_sane(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o, const char *name)
{
int cpu1 = c->cpu_index, cpu2 = o->cpu_index;
return !WARN_ONCE(cpu_to_node(cpu1) != cpu_to_node(cpu2),
return !WARN_ONCE(!topology_same_node(c, o),
"sched: CPU #%d's %s-sibling CPU #%d is not on the same node! "
"[node: %d != %d]. Ignoring dependency.\n",
cpu1, name, cpu2, cpu_to_node(cpu1), cpu_to_node(cpu2));
......@@ -341,17 +349,44 @@ static bool match_llc(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o)
return false;
}
static bool match_mc(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o)
/*
* Unlike the other levels, we do not enforce keeping a
* multicore group inside a NUMA node. If this happens, we will
* discard the MC level of the topology later.
*/
static bool match_die(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o)
{
if (c->phys_proc_id == o->phys_proc_id) {
if (cpu_has(c, X86_FEATURE_AMD_DCM))
return true;
return topology_sane(c, o, "mc");
}
if (c->phys_proc_id == o->phys_proc_id)
return true;
return false;
}
static struct sched_domain_topology_level numa_inside_package_topology[] = {
#ifdef CONFIG_SCHED_SMT
{ cpu_smt_mask, cpu_smt_flags, SD_INIT_NAME(SMT) },
#endif
#ifdef CONFIG_SCHED_MC
{ cpu_coregroup_mask, cpu_core_flags, SD_INIT_NAME(MC) },
#endif
{ NULL, },
};
/*
* set_sched_topology() sets the topology internal to a CPU. The
* NUMA topologies are layered on top of it to build the full
* system topology.
*
* If NUMA nodes are observed to occur within a CPU package, this
* function should be called. It forces the sched domain code to
* only use the SMT level for the CPU portion of the topology.
* This essentially falls back to relying on NUMA information
* from the SRAT table to describe the entire system topology
* (except for hyperthreads).
*/
static void primarily_use_numa_for_topology(void)
{
set_sched_topology(numa_inside_package_topology);
}
void set_cpu_sibling_map(int cpu)
{
bool has_smt = smp_num_siblings > 1;
......@@ -388,7 +423,7 @@ void set_cpu_sibling_map(int cpu)
for_each_cpu(i, cpu_sibling_setup_mask) {
o = &cpu_data(i);
if ((i == cpu) || (has_mp && match_mc(c, o))) {
if ((i == cpu) || (has_mp && match_die(c, o))) {
link_mask(core, cpu, i);
/*
......@@ -410,6 +445,8 @@ void set_cpu_sibling_map(int cpu)
} else if (i != cpu && !c->booted_cores)
c->booted_cores = cpu_data(i).booted_cores;
}
if (match_die(c, o) && !topology_same_node(c, o))
primarily_use_numa_for_topology();
}
}
......
arch/x86/mm/fault.c
浏览文件 @ faafcba3
......@@ -3,7 +3,6 @@
* Copyright (C) 2001, 2002 Andi Kleen, SuSE Labs.
* Copyright (C) 2008-2009, Red Hat Inc., Ingo Molnar
*/
#include <linux/magic.h> /* STACK_END_MAGIC */
#include <linux/sched.h> /* test_thread_flag(), ... */
#include <linux/kdebug.h> /* oops_begin/end, ... */
#include <linux/module.h> /* search_exception_table */
......@@ -649,7 +648,6 @@ no_context(struct pt_regs *regs, unsigned long error_code,
unsigned long address, int signal, int si_code)
{
struct task_struct *tsk = current;
unsigned long *stackend;
unsigned long flags;
int sig;
......@@ -709,8 +707,7 @@ no_context(struct pt_regs *regs, unsigned long error_code,
show_fault_oops(regs, error_code, address);
stackend = end_of_stack(tsk);
if (tsk != &init_task && *stackend != STACK_END_MAGIC)
if (task_stack_end_corrupted(tsk))
printk(KERN_EMERG "Thread overran stack, or stack corrupted\n");
tsk->thread.cr2 = address;
......
drivers/cpuidle/cpuidle.c
浏览文件 @ faafcba3
......@@ -223,8 +223,14 @@ void cpuidle_uninstall_idle_handler(void)
{
if (enabled_devices) {
initialized = 0;
kick_all_cpus_sync();
wake_up_all_idle_cpus();
}
/*
* Make sure external observers (such as the scheduler)
* are done looking at pointed idle states.
*/
synchronize_rcu();
}
/**
......@@ -530,11 +536,6 @@ EXPORT_SYMBOL_GPL(cpuidle_register);
#ifdef CONFIG_SMP
static void smp_callback(void *v)
{
/* we already woke the CPU up, nothing more to do */
}
/*
* This function gets called when a part of the kernel has a new latency
* requirement. This means we need to get all processors out of their C-state,
......@@ -544,7 +545,7 @@ static void smp_callback(void *v)
static int cpuidle_latency_notify(struct notifier_block *b,
unsigned long l, void *v)
{
smp_call_function(smp_callback, NULL, 1);
wake_up_all_idle_cpus();
return NOTIFY_OK;
}
......
drivers/gpu/vga/vgaarb.c
浏览文件 @ faafcba3
......@@ -400,7 +400,6 @@ int vga_get(struct pci_dev *pdev, unsigned int rsrc, int interruptible)
}
schedule();
remove_wait_queue(&vga_wait_queue, &wait);
set_current_state(TASK_RUNNING);
}
return rc;
}
......
drivers/md/dm-bufio.c
浏览文件 @ faafcba3
......@@ -720,7 +720,6 @@ static void __wait_for_free_buffer(struct dm_bufio_client *c)
io_schedule();
set_task_state(current, TASK_RUNNING);
remove_wait_queue(&c->free_buffer_wait, &wait);
dm_bufio_lock(c);
......
drivers/parisc/power.c
浏览文件 @ faafcba3
......@@ -121,7 +121,6 @@ static int kpowerswd(void *param)
unsigned long soft_power_reg = (unsigned long) param;
schedule_timeout_interruptible(pwrsw_enabled ? HZ : HZ/POWERSWITCH_POLL_PER_SEC);
__set_current_state(TASK_RUNNING);
if (unlikely(!pwrsw_enabled))
continue;
......
drivers/s390/net/claw.c
浏览文件 @ faafcba3
......@@ -481,7 +481,6 @@ claw_open(struct net_device *dev)
spin_unlock_irqrestore(
get_ccwdev_lock(privptr->channel[i].cdev), saveflags);
schedule();
set_current_state(TASK_RUNNING);
remove_wait_queue(&privptr->channel[i].wait, &wait);
if(rc != 0)
ccw_check_return_code(privptr->channel[i].cdev, rc);
......@@ -828,7 +827,6 @@ claw_release(struct net_device *dev)
spin_unlock_irqrestore(
get_ccwdev_lock(privptr->channel[i].cdev), saveflags);
schedule();
set_current_state(TASK_RUNNING);
remove_wait_queue(&privptr->channel[i].wait, &wait);
if (rc != 0) {
ccw_check_return_code(privptr->channel[i].cdev, rc);
......
drivers/scsi/fcoe/fcoe.c
浏览文件 @ faafcba3
......@@ -1884,7 +1884,6 @@ static int fcoe_percpu_receive_thread(void *arg)
set_current_state(TASK_INTERRUPTIBLE);
spin_unlock_bh(&p->fcoe_rx_list.lock);
schedule();
set_current_state(TASK_RUNNING);
goto retry;
}
......
drivers/scsi/qla2xxx/qla_os.c
浏览文件 @ faafcba3
......@@ -4875,7 +4875,6 @@ qla2x00_do_dpc(void *data)
"DPC handler sleeping.\n");
schedule();
__set_current_state(TASK_RUNNING);
if (!base_vha->flags.init_done || ha->flags.mbox_busy)
goto end_loop;
......
drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
浏览文件 @ faafcba3
......@@ -3215,7 +3215,6 @@ kiblnd_connd (void *arg)
schedule_timeout(timeout);
set_current_state(TASK_RUNNING);
remove_wait_queue(&kiblnd_data.kib_connd_waitq, &wait);
spin_lock_irqsave(&kiblnd_data.kib_connd_lock, flags);
}
......@@ -3432,7 +3431,6 @@ kiblnd_scheduler(void *arg)
busy_loops = 0;
remove_wait_queue(&sched->ibs_waitq, &wait);
set_current_state(TASK_RUNNING);
spin_lock_irqsave(&sched->ibs_lock, flags);
}
......@@ -3507,7 +3505,6 @@ kiblnd_failover_thread(void *arg)
rc = schedule_timeout(long_sleep ? cfs_time_seconds(10) :
cfs_time_seconds(1));
set_current_state(TASK_RUNNING);
remove_wait_queue(&kiblnd_data.kib_failover_waitq, &wait);
write_lock_irqsave(glock, flags);
......
drivers/staging/lustre/lnet/klnds/socklnd/socklnd_cb.c
浏览文件 @ faafcba3
......@@ -2232,7 +2232,6 @@ ksocknal_connd (void *arg)
nloops = 0;
schedule_timeout(timeout);
set_current_state(TASK_RUNNING);
remove_wait_queue(&ksocknal_data.ksnd_connd_waitq, &wait);
spin_lock_bh(connd_lock);
}
......
drivers/staging/lustre/lustre/libcfs/fail.c
浏览文件 @ faafcba3
......@@ -131,7 +131,6 @@ int __cfs_fail_timeout_set(__u32 id, __u32 value, int ms, int set)
id, ms);
set_current_state(TASK_UNINTERRUPTIBLE);
schedule_timeout(cfs_time_seconds(ms) / 1000);
set_current_state(TASK_RUNNING);
CERROR("cfs_fail_timeout id %x awake\n", id);
}
return ret;
......
drivers/tty/bfin_jtag_comm.c
浏览文件 @ faafcba3
......@@ -77,7 +77,6 @@ bfin_jc_emudat_manager(void *arg)
pr_debug("waiting for readers\n");
__set_current_state(TASK_UNINTERRUPTIBLE);
schedule();
__set_current_state(TASK_RUNNING);
continue;
}
......
fs/afs/vlocation.c
浏览文件 @ faafcba3
......@@ -130,7 +130,6 @@ static int afs_vlocation_access_vl_by_id(struct afs_vlocation *vl,
/* second+ BUSY - sleep a little bit */
set_current_state(TASK_UNINTERRUPTIBLE);
schedule_timeout(1);
__set_current_state(TASK_RUNNING);
}
continue;
}
......
fs/jfs/jfs_logmgr.c
浏览文件 @ faafcba3
......@@ -1585,7 +1585,6 @@ void jfs_flush_journal(struct jfs_log *log, int wait)
set_current_state(TASK_UNINTERRUPTIBLE);
LOGGC_UNLOCK(log);
schedule();
__set_current_state(TASK_RUNNING);
LOGGC_LOCK(log);
remove_wait_queue(&target->gcwait, &__wait);
}
......@@ -2359,7 +2358,6 @@ int jfsIOWait(void *arg)
set_current_state(TASK_INTERRUPTIBLE);
spin_unlock_irq(&log_redrive_lock);
schedule();
__set_current_state(TASK_RUNNING);
}
} while (!kthread_should_stop());
......
fs/jfs/jfs_txnmgr.c
浏览文件 @ faafcba3
......@@ -136,7 +136,6 @@ static inline void TXN_SLEEP_DROP_LOCK(wait_queue_head_t * event)
set_current_state(TASK_UNINTERRUPTIBLE);
TXN_UNLOCK();
io_schedule();
__set_current_state(TASK_RUNNING);
remove_wait_queue(event, &wait);
}
......@@ -2808,7 +2807,6 @@ int jfs_lazycommit(void *arg)
set_current_state(TASK_INTERRUPTIBLE);
LAZY_UNLOCK(flags);
schedule();
__set_current_state(TASK_RUNNING);
remove_wait_queue(&jfs_commit_thread_wait, &wq);
}
} while (!kthread_should_stop());
......@@ -2996,7 +2994,6 @@ int jfs_sync(void *arg)
set_current_state(TASK_INTERRUPTIBLE);
TXN_UNLOCK();
schedule();
__set_current_state(TASK_RUNNING);
}
} while (!kthread_should_stop());
......
fs/nfs/blocklayout/rpc_pipefs.c
浏览文件 @ faafcba3
......@@ -92,7 +92,6 @@ bl_resolve_deviceid(struct nfs_server *server, struct pnfs_block_volume *b,
set_current_state(TASK_UNINTERRUPTIBLE);
schedule();
__set_current_state(TASK_RUNNING);
remove_wait_queue(&nn->bl_wq, &wq);
if (reply->status != BL_DEVICE_REQUEST_PROC) {
......
fs/nfsd/nfs4recover.c
浏览文件 @ faafcba3
......@@ -675,7 +675,6 @@ __cld_pipe_upcall(struct rpc_pipe *pipe, struct cld_msg *cmsg)
}
schedule();
set_current_state(TASK_RUNNING);
if (msg.errno < 0)
ret = msg.errno;
......
include/asm-generic/cputime_jiffies.h
浏览文件 @ faafcba3
......@@ -3,6 +3,8 @@
typedef unsigned long __nocast cputime_t;
#define cmpxchg_cputime(ptr, old, new) cmpxchg(ptr, old, new)
#define cputime_one_jiffy jiffies_to_cputime(1)
#define cputime_to_jiffies(__ct) (__force unsigned long)(__ct)
#define cputime_to_scaled(__ct) (__ct)
......
include/asm-generic/cputime_nsecs.h
浏览文件 @ faafcba3
......@@ -21,6 +21,8 @@
typedef u64 __nocast cputime_t;
typedef u64 __nocast cputime64_t;
#define cmpxchg_cputime(ptr, old, new) cmpxchg64(ptr, old, new)
#define cputime_one_jiffy jiffies_to_cputime(1)
#define cputime_div(__ct, divisor) div_u64((__force u64)__ct, divisor)
......
include/linux/sched.h
浏览文件 @ faafcba3
......@@ -57,6 +57,7 @@ struct sched_param {
#include <linux/llist.h>
#include <linux/uidgid.h>
#include <linux/gfp.h>
#include <linux/magic.h>
#include <asm/processor.h>
......@@ -646,6 +647,7 @@ struct signal_struct {
* Live threads maintain their own counters and add to these
* in __exit_signal, except for the group leader.
*/
seqlock_t stats_lock;
cputime_t utime, stime, cutime, cstime;
cputime_t gtime;
cputime_t cgtime;
......@@ -1024,6 +1026,7 @@ struct sched_domain_topology_level {
extern struct sched_domain_topology_level *sched_domain_topology;
extern void set_sched_topology(struct sched_domain_topology_level *tl);
extern void wake_up_if_idle(int cpu);
#ifdef CONFIG_SCHED_DEBUG
# define SD_INIT_NAME(type) .name = #type
......@@ -2647,6 +2650,8 @@ static inline unsigned long *end_of_stack(struct task_struct *p)
}
#endif
#define task_stack_end_corrupted(task) \
(*(end_of_stack(task)) != STACK_END_MAGIC)
static inline int object_is_on_stack(void *obj)
{
......@@ -2669,6 +2674,7 @@ static inline unsigned long stack_not_used(struct task_struct *p)
return (unsigned long)n - (unsigned long)end_of_stack(p);
}
#endif
extern void set_task_stack_end_magic(struct task_struct *tsk);
/* set thread flags in other task's structures
* - see asm/thread_info.h for TIF_xxxx flags available
......
include/linux/seqlock.h
浏览文件 @ faafcba3
......@@ -456,4 +456,23 @@ read_sequnlock_excl_irqrestore(seqlock_t *sl, unsigned long flags)
spin_unlock_irqrestore(&sl->lock, flags);
}
static inline unsigned long
read_seqbegin_or_lock_irqsave(seqlock_t *lock, int *seq)
{
unsigned long flags = 0;
if (!(*seq & 1)) /* Even */
*seq = read_seqbegin(lock);
else /* Odd */
read_seqlock_excl_irqsave(lock, flags);
return flags;
}
static inline void
done_seqretry_irqrestore(seqlock_t *lock, int seq, unsigned long flags)
{
if (seq & 1)
read_sequnlock_excl_irqrestore(lock, flags);
}
#endif /* __LINUX_SEQLOCK_H */
include/linux/smp.h
浏览文件 @ faafcba3
......@@ -100,6 +100,7 @@ int smp_call_function_any(const struct cpumask *mask,
smp_call_func_t func, void *info, int wait);
void kick_all_cpus_sync(void);
void wake_up_all_idle_cpus(void);
/*
* Generic and arch helpers
......@@ -148,6 +149,7 @@ smp_call_function_any(const struct cpumask *mask, smp_call_func_t func,
}
static inline void kick_all_cpus_sync(void) { }
static inline void wake_up_all_idle_cpus(void) { }
#endif /* !SMP */
......
include/linux/wait.h
浏览文件 @ faafcba3
......@@ -281,9 +281,11 @@ do { \
* wake_up() has to be called after changing any variable that could
* change the result of the wait condition.
*
* The function returns 0 if the @timeout elapsed, or the remaining
* jiffies (at least 1) if the @condition evaluated to %true before
* the @timeout elapsed.
* Returns:
* 0 if the @condition evaluated to %false after the @timeout elapsed,
* 1 if the @condition evaluated to %true after the @timeout elapsed,
* or the remaining jiffies (at least 1) if the @condition evaluated
* to %true before the @timeout elapsed.
*/
#define wait_event_timeout(wq, condition, timeout) \
({ \
......@@ -364,9 +366,11 @@ do { \
* change the result of the wait condition.
*
* Returns:
* 0 if the @timeout elapsed, -%ERESTARTSYS if it was interrupted by
* a signal, or the remaining jiffies (at least 1) if the @condition
* evaluated to %true before the @timeout elapsed.
* 0 if the @condition evaluated to %false after the @timeout elapsed,
* 1 if the @condition evaluated to %true after the @timeout elapsed,
* the remaining jiffies (at least 1) if the @condition evaluated
* to %true before the @timeout elapsed, or -%ERESTARTSYS if it was
* interrupted by a signal.
*/
#define wait_event_interruptible_timeout(wq, condition, timeout) \
({ \
......
init/main.c
浏览文件 @ faafcba3
......@@ -508,6 +508,7 @@ asmlinkage __visible void __init start_kernel(void)
* lockdep hash:
*/
lockdep_init();
set_task_stack_end_magic(&init_task);
smp_setup_processor_id();
debug_objects_early_init();
......
kernel/exit.c
浏览文件 @ faafcba3
......@@ -115,32 +115,33 @@ static void __exit_signal(struct task_struct *tsk)
if (tsk == sig->curr_target)
sig->curr_target = next_thread(tsk);
/*
* Accumulate here the counters for all threads but the
* group leader as they die, so they can be added into
* the process-wide totals when those are taken.
* The group leader stays around as a zombie as long
* as there are other threads. When it gets reaped,
* the exit.c code will add its counts into these totals.
* We won't ever get here for the group leader, since it
* will have been the last reference on the signal_struct.
*/
task_cputime(tsk, &utime, &stime);
sig->utime += utime;
sig->stime += stime;
sig->gtime += task_gtime(tsk);
sig->min_flt += tsk->min_flt;
sig->maj_flt += tsk->maj_flt;
sig->nvcsw += tsk->nvcsw;
sig->nivcsw += tsk->nivcsw;
sig->inblock += task_io_get_inblock(tsk);
sig->oublock += task_io_get_oublock(tsk);
task_io_accounting_add(&sig->ioac, &tsk->ioac);
sig->sum_sched_runtime += tsk->se.sum_exec_runtime;
}
/*
* Accumulate here the counters for all threads but the group leader
* as they die, so they can be added into the process-wide totals
* when those are taken. The group leader stays around as a zombie as
* long as there are other threads. When it gets reaped, the exit.c
* code will add its counts into these totals. We won't ever get here
* for the group leader, since it will have been the last reference on
* the signal_struct.
*/
task_cputime(tsk, &utime, &stime);
write_seqlock(&sig->stats_lock);
sig->utime += utime;
sig->stime += stime;
sig->gtime += task_gtime(tsk);
sig->min_flt += tsk->min_flt;
sig->maj_flt += tsk->maj_flt;
sig->nvcsw += tsk->nvcsw;
sig->nivcsw += tsk->nivcsw;
sig->inblock += task_io_get_inblock(tsk);
sig->oublock += task_io_get_oublock(tsk);
task_io_accounting_add(&sig->ioac, &tsk->ioac);
sig->sum_sched_runtime += tsk->se.sum_exec_runtime;
sig->nr_threads--;
__unhash_process(tsk, group_dead);
write_sequnlock(&sig->stats_lock);
/*
* Do this under ->siglock, we can race with another thread
......@@ -1046,6 +1047,7 @@ static int wait_task_zombie(struct wait_opts *wo, struct task_struct *p)
spin_lock_irq(&p->real_parent->sighand->siglock);
psig = p->real_parent->signal;
sig = p->signal;
write_seqlock(&psig->stats_lock);
psig->cutime += tgutime + sig->cutime;
psig->cstime += tgstime + sig->cstime;
psig->cgtime += task_gtime(p) + sig->gtime + sig->cgtime;
......@@ -1068,6 +1070,7 @@ static int wait_task_zombie(struct wait_opts *wo, struct task_struct *p)
psig->cmaxrss = maxrss;
task_io_accounting_add(&psig->ioac, &p->ioac);
task_io_accounting_add(&psig->ioac, &sig->ioac);
write_sequnlock(&psig->stats_lock);
spin_unlock_irq(&p->real_parent->sighand->siglock);
}
......
kernel/fork.c
浏览文件 @ faafcba3
......@@ -294,11 +294,18 @@ int __weak arch_dup_task_struct(struct task_struct *dst,
return 0;
}
void set_task_stack_end_magic(struct task_struct *tsk)
{
unsigned long *stackend;
stackend = end_of_stack(tsk);
*stackend = STACK_END_MAGIC; /* for overflow detection */
}
static struct task_struct *dup_task_struct(struct task_struct *orig)
{
struct task_struct *tsk;
struct thread_info *ti;
unsigned long *stackend;
int node = tsk_fork_get_node(orig);
int err;
......@@ -328,8 +335,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig)
setup_thread_stack(tsk, orig);
clear_user_return_notifier(tsk);
clear_tsk_need_resched(tsk);
stackend = end_of_stack(tsk);
*stackend = STACK_END_MAGIC; /* for overflow detection */
set_task_stack_end_magic(tsk);
#ifdef CONFIG_CC_STACKPROTECTOR
tsk->stack_canary = get_random_int();
......@@ -1067,6 +1073,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk)
sig->curr_target = tsk;
init_sigpending(&sig->shared_pending);
INIT_LIST_HEAD(&sig->posix_timers);
seqlock_init(&sig->stats_lock);
hrtimer_init(&sig->real_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
sig->real_timer.function = it_real_fn;
......
kernel/sched/auto_group.c
浏览文件 @ faafcba3
......@@ -148,11 +148,8 @@ autogroup_move_group(struct task_struct *p, struct autogroup *ag)
if (!ACCESS_ONCE(sysctl_sched_autogroup_enabled))
goto out;
t = p;
do {
for_each_thread(p, t)
sched_move_task(t);
} while_each_thread(p, t);
out:
unlock_task_sighand(p, &flags);
autogroup_kref_put(prev);
......
kernel/sched/core.c
浏览文件 @ faafcba3
此差异已折叠。
kernel/sched/cpudeadline.c
浏览文件 @ faafcba3
......@@ -107,9 +107,7 @@ int cpudl_find(struct cpudl *cp, struct task_struct *p,
int best_cpu = -1;
const struct sched_dl_entity *dl_se = &p->dl;
if (later_mask && cpumask_and(later_mask, cp->free_cpus,
&p->cpus_allowed) && cpumask_and(later_mask,
later_mask, cpu_active_mask)) {
if (later_mask && cpumask_and(later_mask, later_mask, cp->free_cpus)) {
best_cpu = cpumask_any(later_mask);
goto out;
} else if (cpumask_test_cpu(cpudl_maximum(cp), &p->cpus_allowed) &&
......
kernel/sched/cputime.c
浏览文件 @ faafcba3
......@@ -288,24 +288,29 @@ void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)
struct signal_struct *sig = tsk->signal;
cputime_t utime, stime;
struct task_struct *t;
times->utime = sig->utime;
times->stime = sig->stime;
times->sum_exec_runtime = sig->sum_sched_runtime;
unsigned int seq, nextseq;
unsigned long flags;
rcu_read_lock();
/* make sure we can trust tsk->thread_group list */
if (!likely(pid_alive(tsk)))
goto out;
t = tsk;
/* Attempt a lockless read on the first round. */
nextseq = 0;
do {
task_cputime(t, &utime, &stime);
times->utime += utime;
times->stime += stime;
times->sum_exec_runtime += task_sched_runtime(t);
} while_each_thread(tsk, t);
out:
seq = nextseq;
flags = read_seqbegin_or_lock_irqsave(&sig->stats_lock, &seq);
times->utime = sig->utime;
times->stime = sig->stime;
times->sum_exec_runtime = sig->sum_sched_runtime;
for_each_thread(tsk, t) {
task_cputime(t, &utime, &stime);
times->utime += utime;
times->stime += stime;
times->sum_exec_runtime += task_sched_runtime(t);
}
/* If lockless access failed, take the lock. */
nextseq = 1;
} while (need_seqretry(&sig->stats_lock, seq));
done_seqretry_irqrestore(&sig->stats_lock, seq, flags);
rcu_read_unlock();
}
......@@ -549,6 +554,23 @@ static cputime_t scale_stime(u64 stime, u64 rtime, u64 total)
return (__force cputime_t) scaled;
}
/*
* Atomically advance counter to the new value. Interrupts, vcpu
* scheduling, and scaling inaccuracies can cause cputime_advance
* to be occasionally called with a new value smaller than counter.
* Let's enforce atomicity.
*
* Normally a caller will only go through this loop once, or not
* at all in case a previous caller updated counter the same jiffy.
*/
static void cputime_advance(cputime_t *counter, cputime_t new)
{
cputime_t old;
while (new > (old = ACCESS_ONCE(*counter)))
cmpxchg_cputime(counter, old, new);
}
/*
* Adjust tick based cputime random precision against scheduler
* runtime accounting.
......@@ -594,13 +616,8 @@ static void cputime_adjust(struct task_cputime *curr,
utime = rtime - stime;
}
/*
* If the tick based count grows faster than the scheduler one,
* the result of the scaling may go backward.
* Let's enforce monotonicity.
*/
prev->stime = max(prev->stime, stime);
prev->utime = max(prev->utime, utime);
cputime_advance(&prev->stime, stime);
cputime_advance(&prev->utime, utime);
out:
*ut = prev->utime;
......@@ -617,9 +634,6 @@ void task_cputime_adjusted(struct task_struct *p, cputime_t *ut, cputime_t *st)
cputime_adjust(&cputime, &p->prev_cputime, ut, st);
}
/*
* Must be called with siglock held.
*/
void thread_group_cputime_adjusted(struct task_struct *p, cputime_t *ut, cputime_t *st)
{
struct task_cputime cputime;
......
kernel/sched/deadline.c
浏览文件 @ faafcba3
......@@ -530,7 +530,7 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
update_rq_clock(rq);
dl_se->dl_throttled = 0;
dl_se->dl_yielded = 0;
if (p->on_rq) {
if (task_on_rq_queued(p)) {
enqueue_task_dl(rq, p, ENQUEUE_REPLENISH);
if (task_has_dl_policy(rq->curr))
check_preempt_curr_dl(rq, p, 0);
......@@ -997,10 +997,7 @@ static void check_preempt_curr_dl(struct rq *rq, struct task_struct *p,
#ifdef CONFIG_SCHED_HRTICK
static void start_hrtick_dl(struct rq *rq, struct task_struct *p)
{
s64 delta = p->dl.dl_runtime - p->dl.runtime;
if (delta > 10000)
hrtick_start(rq, p->dl.runtime);
hrtick_start(rq, p->dl.runtime);
}
#endif
......@@ -1030,7 +1027,7 @@ struct task_struct *pick_next_task_dl(struct rq *rq, struct task_struct *prev)
* means a stop task can slip in, in which case we need to
* re-start task selection.
*/
if (rq->stop && rq->stop->on_rq)
if (rq->stop && task_on_rq_queued(rq->stop))
return RETRY_TASK;
}
......@@ -1124,10 +1121,8 @@ static void set_curr_task_dl(struct rq *rq)
static int pick_dl_task(struct rq *rq, struct task_struct *p, int cpu)
{
if (!task_running(rq, p) &&
(cpu < 0 || cpumask_test_cpu(cpu, &p->cpus_allowed)) &&
(p->nr_cpus_allowed > 1))
cpumask_test_cpu(cpu, tsk_cpus_allowed(p)))
return 1;
return 0;
}
......@@ -1169,6 +1164,13 @@ static int find_later_rq(struct task_struct *task)
if (task->nr_cpus_allowed == 1)
return -1;
/*
* We have to consider system topology and task affinity
* first, then we can look for a suitable cpu.
*/
cpumask_copy(later_mask, task_rq(task)->rd->span);
cpumask_and(later_mask, later_mask, cpu_active_mask);
cpumask_and(later_mask, later_mask, &task->cpus_allowed);
best_cpu = cpudl_find(&task_rq(task)->rd->cpudl,
task, later_mask);
if (best_cpu == -1)
......@@ -1257,7 +1259,8 @@ static struct rq *find_lock_later_rq(struct task_struct *task, struct rq *rq)
if (unlikely(task_rq(task) != rq ||
!cpumask_test_cpu(later_rq->cpu,
&task->cpus_allowed) ||
task_running(rq, task) || !task->on_rq)) {
task_running(rq, task) ||
!task_on_rq_queued(task))) {
double_unlock_balance(rq, later_rq);
later_rq = NULL;
break;
......@@ -1296,7 +1299,7 @@ static struct task_struct *pick_next_pushable_dl_task(struct rq *rq)
BUG_ON(task_current(rq, p));
BUG_ON(p->nr_cpus_allowed <= 1);
BUG_ON(!p->on_rq);
BUG_ON(!task_on_rq_queued(p));
BUG_ON(!dl_task(p));
return p;
......@@ -1443,7 +1446,7 @@ static int pull_dl_task(struct rq *this_rq)
dl_time_before(p->dl.deadline,
this_rq->dl.earliest_dl.curr))) {
WARN_ON(p == src_rq->curr);
WARN_ON(!p->on_rq);
WARN_ON(!task_on_rq_queued(p));
/*
* Then we pull iff p has actually an earlier
......@@ -1569,6 +1572,8 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
if (hrtimer_active(&p->dl.dl_timer) && !dl_policy(p->policy))
hrtimer_try_to_cancel(&p->dl.dl_timer);
__dl_clear_params(p);
#ifdef CONFIG_SMP
/*
* Since this might be the only -deadline task on the rq,
......@@ -1596,7 +1601,7 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
if (unlikely(p->dl.dl_throttled))
return;
if (p->on_rq && rq->curr != p) {
if (task_on_rq_queued(p) && rq->curr != p) {
#ifdef CONFIG_SMP
if (rq->dl.overloaded && push_dl_task(rq) && rq != task_rq(p))
/* Only reschedule if pushing failed */
......@@ -1614,7 +1619,7 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
static void prio_changed_dl(struct rq *rq, struct task_struct *p,
int oldprio)
{
if (p->on_rq || rq->curr == p) {
if (task_on_rq_queued(p) || rq->curr == p) {
#ifdef CONFIG_SMP
/*
* This might be too much, but unfortunately
......
kernel/sched/debug.c
浏览文件 @ faafcba3
......@@ -150,7 +150,6 @@ print_task(struct seq_file *m, struct rq *rq, struct task_struct *p)
static void print_rq(struct seq_file *m, struct rq *rq, int rq_cpu)
{
struct task_struct *g, *p;
unsigned long flags;
SEQ_printf(m,
"\nrunnable tasks:\n"
......@@ -159,16 +158,14 @@ static void print_rq(struct seq_file *m, struct rq *rq, int rq_cpu)
"------------------------------------------------------"
"----------------------------------------------------\n");
read_lock_irqsave(&tasklist_lock, flags);
do_each_thread(g, p) {
rcu_read_lock();
for_each_process_thread(g, p) {
if (task_cpu(p) != rq_cpu)
continue;
print_task(m, rq, p);
} while_each_thread(g, p);
read_unlock_irqrestore(&tasklist_lock, flags);
}
rcu_read_unlock();
}
void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
......@@ -333,9 +330,7 @@ do { \
print_cfs_stats(m, cpu);
print_rt_stats(m, cpu);
rcu_read_lock();
print_rq(m, rq, cpu);
rcu_read_unlock();
spin_unlock_irqrestore(&sched_debug_lock, flags);
SEQ_printf(m, "\n");
}
......
kernel/sched/fair.c
浏览文件 @ faafcba3
此差异已折叠。
kernel/sched/idle.c
浏览文件 @ faafcba3
......@@ -147,6 +147,9 @@ static void cpuidle_idle_call(void)
clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ENTER, &dev->cpu))
goto use_default;
/* Take note of the planned idle state. */
idle_set_state(this_rq(), &drv->states[next_state]);
/*
* Enter the idle state previously returned by the governor decision.
* This function will block until an interrupt occurs and will take
......@@ -154,6 +157,9 @@ static void cpuidle_idle_call(void)
*/
entered_state = cpuidle_enter(drv, dev, next_state);
/* The cpu is no longer idle or about to enter idle. */
idle_set_state(this_rq(), NULL);
if (broadcast)
clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_EXIT, &dev->cpu);
......
kernel/sched/rt.c
浏览文件 @ faafcba3
......@@ -1448,7 +1448,7 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev)
* means a dl or stop task can slip in, in which case we need
* to re-start task selection.
*/
if (unlikely((rq->stop && rq->stop->on_rq) ||
if (unlikely((rq->stop && task_on_rq_queued(rq->stop)) ||
rq->dl.dl_nr_running))
return RETRY_TASK;
}
......@@ -1468,8 +1468,7 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev)
p = _pick_next_task_rt(rq);
/* The running task is never eligible for pushing */
if (p)
dequeue_pushable_task(rq, p);
dequeue_pushable_task(rq, p);
set_post_schedule(rq);
......@@ -1624,7 +1623,7 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)
!cpumask_test_cpu(lowest_rq->cpu,
tsk_cpus_allowed(task)) ||
task_running(rq, task) ||
!task->on_rq)) {
!task_on_rq_queued(task))) {
double_unlock_balance(rq, lowest_rq);
lowest_rq = NULL;
......@@ -1658,7 +1657,7 @@ static struct task_struct *pick_next_pushable_task(struct rq *rq)
BUG_ON(task_current(rq, p));
BUG_ON(p->nr_cpus_allowed <= 1);
BUG_ON(!p->on_rq);
BUG_ON(!task_on_rq_queued(p));
BUG_ON(!rt_task(p));
return p;
......@@ -1809,7 +1808,7 @@ static int pull_rt_task(struct rq *this_rq)
*/
if (p && (p->prio < this_rq->rt.highest_prio.curr)) {
WARN_ON(p == src_rq->curr);
WARN_ON(!p->on_rq);
WARN_ON(!task_on_rq_queued(p));
/*
* There's a chance that p is higher in priority
......@@ -1870,7 +1869,7 @@ static void set_cpus_allowed_rt(struct task_struct *p,
BUG_ON(!rt_task(p));
if (!p->on_rq)
if (!task_on_rq_queued(p))
return;
weight = cpumask_weight(new_mask);
......@@ -1936,7 +1935,7 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)
* we may need to handle the pulling of RT tasks
* now.
*/
if (!p->on_rq || rq->rt.rt_nr_running)
if (!task_on_rq_queued(p) || rq->rt.rt_nr_running)
return;
if (pull_rt_task(rq))
......@@ -1970,7 +1969,7 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)
* If that current running task is also an RT task
* then see if we can move to another run queue.
*/
if (p->on_rq && rq->curr != p) {
if (task_on_rq_queued(p) && rq->curr != p) {
#ifdef CONFIG_SMP
if (p->nr_cpus_allowed > 1 && rq->rt.overloaded &&
/* Don't resched if we changed runqueues */
......@@ -1989,7 +1988,7 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)
static void
prio_changed_rt(struct rq *rq, struct task_struct *p, int oldprio)
{
if (!p->on_rq)
if (!task_on_rq_queued(p))
return;
if (rq->curr == p) {
......@@ -2073,7 +2072,7 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued)
for_each_sched_rt_entity(rt_se) {
if (rt_se->run_list.prev != rt_se->run_list.next) {
requeue_task_rt(rq, p, 0);
set_tsk_need_resched(p);
resched_curr(rq);
return;
}
}
......
kernel/sched/sched.h
浏览文件 @ faafcba3
......@@ -14,6 +14,11 @@
#include "cpuacct.h"
struct rq;
struct cpuidle_state;
/* task_struct::on_rq states: */
#define TASK_ON_RQ_QUEUED 1
#define TASK_ON_RQ_MIGRATING 2
extern __read_mostly int scheduler_running;
......@@ -126,6 +131,9 @@ struct rt_bandwidth {
u64 rt_runtime;
struct hrtimer rt_period_timer;
};
void __dl_clear_params(struct task_struct *p);
/*
* To keep the bandwidth of -deadline tasks and groups under control
* we need some place where:
......@@ -184,7 +192,7 @@ struct cfs_bandwidth {
raw_spinlock_t lock;
ktime_t period;
u64 quota, runtime;
s64 hierarchal_quota;
s64 hierarchical_quota;
u64 runtime_expires;
int idle, timer_active;
......@@ -636,6 +644,11 @@ struct rq {
#ifdef CONFIG_SMP
struct llist_head wake_list;
#endif
#ifdef CONFIG_CPU_IDLE
/* Must be inspected within a rcu lock section */
struct cpuidle_state *idle_state;
#endif
};
static inline int cpu_of(struct rq *rq)
......@@ -647,7 +660,7 @@ static inline int cpu_of(struct rq *rq)
#endif
}
DECLARE_PER_CPU(struct rq, runqueues);
DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
#define cpu_rq(cpu) (&per_cpu(runqueues, (cpu)))
#define this_rq() (&__get_cpu_var(runqueues))
......@@ -942,6 +955,15 @@ static inline int task_running(struct rq *rq, struct task_struct *p)
#endif
}
static inline int task_on_rq_queued(struct task_struct *p)
{
return p->on_rq == TASK_ON_RQ_QUEUED;
}
static inline int task_on_rq_migrating(struct task_struct *p)
{
return p->on_rq == TASK_ON_RQ_MIGRATING;
}
#ifndef prepare_arch_switch
# define prepare_arch_switch(next) do { } while (0)
......@@ -953,7 +975,6 @@ static inline int task_running(struct rq *rq, struct task_struct *p)
# define finish_arch_post_lock_switch() do { } while (0)
#endif
#ifndef __ARCH_WANT_UNLOCKED_CTXSW
static inline void prepare_lock_switch(struct rq *rq, struct task_struct *next)
{
#ifdef CONFIG_SMP
......@@ -991,35 +1012,6 @@ static inline void finish_lock_switch(struct rq *rq, struct task_struct *prev)
raw_spin_unlock_irq(&rq->lock);
}
#else /* __ARCH_WANT_UNLOCKED_CTXSW */
static inline void prepare_lock_switch(struct rq *rq, struct task_struct *next)
{
#ifdef CONFIG_SMP
/*
* We can optimise this out completely for !SMP, because the
* SMP rebalancing from interrupt is the only thing that cares
* here.
*/
next->on_cpu = 1;
#endif
raw_spin_unlock(&rq->lock);
}
static inline void finish_lock_switch(struct rq *rq, struct task_struct *prev)
{
#ifdef CONFIG_SMP
/*
* After ->on_cpu is cleared, the task can be moved to a different CPU.
* We must ensure this doesn't happen until the switch is completely
* finished.
*/
smp_wmb();
prev->on_cpu = 0;
#endif
local_irq_enable();
}
#endif /* __ARCH_WANT_UNLOCKED_CTXSW */
/*
* wake flags
*/
......@@ -1180,6 +1172,30 @@ static inline void idle_exit_fair(struct rq *rq) { }
#endif
#ifdef CONFIG_CPU_IDLE
static inline void idle_set_state(struct rq *rq,
struct cpuidle_state *idle_state)
{
rq->idle_state = idle_state;
}
static inline struct cpuidle_state *idle_get_state(struct rq *rq)
{
WARN_ON(!rcu_read_lock_held());
return rq->idle_state;
}
#else
static inline void idle_set_state(struct rq *rq,
struct cpuidle_state *idle_state)
{
}
static inline struct cpuidle_state *idle_get_state(struct rq *rq)
{
return NULL;
}
#endif
extern void sysrq_sched_debug_show(void);
extern void sched_init_granularity(void);
extern void update_max_interval(void);
......
kernel/sched/stop_task.c
浏览文件 @ faafcba3
......@@ -28,7 +28,7 @@ pick_next_task_stop(struct rq *rq, struct task_struct *prev)
{
struct task_struct *stop = rq->stop;
if (!stop || !stop->on_rq)
if (!stop || !task_on_rq_queued(stop))
return NULL;
put_prev_task(rq, prev);
......
kernel/smp.c
浏览文件 @ faafcba3
......@@ -13,6 +13,7 @@
#include <linux/gfp.h>
#include <linux/smp.h>
#include <linux/cpu.h>
#include <linux/sched.h>
#include "smpboot.h"
......@@ -699,3 +700,24 @@ void kick_all_cpus_sync(void)
smp_call_function(do_nothing, NULL, 1);
}
EXPORT_SYMBOL_GPL(kick_all_cpus_sync);
/**
* wake_up_all_idle_cpus - break all cpus out of idle
* wake_up_all_idle_cpus try to break all cpus which is in idle state even
* including idle polling cpus, for non-idle cpus, we will do nothing
* for them.
*/
void wake_up_all_idle_cpus(void)
{
int cpu;
preempt_disable();
for_each_online_cpu(cpu) {
if (cpu == smp_processor_id())
continue;
wake_up_if_idle(cpu);
}
preempt_enable();
}
EXPORT_SYMBOL_GPL(wake_up_all_idle_cpus);
kernel/sys.c
浏览文件 @ faafcba3
......@@ -869,11 +869,9 @@ void do_sys_times(struct tms *tms)
{
cputime_t tgutime, tgstime, cutime, cstime;
spin_lock_irq(&current->sighand->siglock);
thread_group_cputime_adjusted(current, &tgutime, &tgstime);
cutime = current->signal->cutime;
cstime = current->signal->cstime;
spin_unlock_irq(&current->sighand->siglock);
tms->tms_utime = cputime_to_clock_t(tgutime);
tms->tms_stime = cputime_to_clock_t(tgstime);
tms->tms_cutime = cputime_to_clock_t(cutime);
......
kernel/time/hrtimer.c
浏览文件 @ faafcba3
......@@ -1776,7 +1776,6 @@ schedule_hrtimeout_range_clock(ktime_t *expires, unsigned long delta,
*/
if (!expires) {
schedule();
__set_current_state(TASK_RUNNING);
return -EINTR;
}
......
kernel/time/posix-cpu-timers.c
浏览文件 @ faafcba3
......@@ -272,22 +272,8 @@ static int posix_cpu_clock_get_task(struct task_struct *tsk,
if (same_thread_group(tsk, current))
err = cpu_clock_sample(which_clock, tsk, &rtn);
} else {
unsigned long flags;
struct sighand_struct *sighand;
/*
* while_each_thread() is not yet entirely RCU safe,
* keep locking the group while sampling process
* clock for now.
*/
sighand = lock_task_sighand(tsk, &flags);
if (!sighand)
return err;
if (tsk == current || thread_group_leader(tsk))
err = cpu_clock_sample_group(which_clock, tsk, &rtn);
unlock_task_sighand(tsk, &flags);
}
if (!err)
......
kernel/trace/ring_buffer_benchmark.c
浏览文件 @ faafcba3
......@@ -205,7 +205,6 @@ static void ring_buffer_consumer(void)
break;
schedule();
__set_current_state(TASK_RUNNING);
}
reader_finish = 0;
complete(&read_done);
......@@ -379,7 +378,6 @@ static int ring_buffer_consumer_thread(void *arg)
break;
schedule();
__set_current_state(TASK_RUNNING);
}
__set_current_state(TASK_RUNNING);
......@@ -407,7 +405,6 @@ static int ring_buffer_producer_thread(void *arg)
trace_printk("Sleeping for 10 secs\n");
set_current_state(TASK_INTERRUPTIBLE);
schedule_timeout(HZ * SLEEP_TIME);
__set_current_state(TASK_RUNNING);
}
if (kill_test)
......
kernel/trace/trace_stack.c
浏览文件 @ faafcba3
......@@ -13,7 +13,6 @@
#include <linux/sysctl.h>
#include <linux/init.h>
#include <linux/fs.h>
#include <linux/magic.h>
#include <asm/setup.h>
......@@ -171,8 +170,7 @@ check_stack(unsigned long ip, unsigned long *stack)
i++;
}
if ((current != &init_task &&
*(end_of_stack(current)) != STACK_END_MAGIC)) {
if (task_stack_end_corrupted(current)) {
print_max_stack();
BUG();
}
......
lib/Kconfig.debug
浏览文件 @ faafcba3
......@@ -824,6 +824,18 @@ config SCHEDSTATS
application, you can say N to avoid the very slight overhead
this adds.
config SCHED_STACK_END_CHECK
bool "Detect stack corruption on calls to schedule()"
depends on DEBUG_KERNEL
default n
help
This option checks for a stack overrun on calls to schedule().
If the stack end location is found to be over written always panic as
the content of the corrupted region can no longer be trusted.
This is to ensure no erroneous behaviour occurs which could result in
data corruption or a sporadic crash at a later stage once the region
is examined. The runtime overhead introduced is minimal.
config TIMER_STATS
bool "Collect kernel timers statistics"
depends on DEBUG_KERNEL && PROC_FS
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或