1. 26 5月, 2011 1 次提交
    • A
      perf symbols: Handle /proc/sys/kernel/kptr_restrict · ec80fde7
      Arnaldo Carvalho de Melo 提交于
      Perf uses /proc/modules to figure out where kernel modules are loaded.
      
      With the advent of kptr_restrict, non root users get zeroes for all module
      start addresses.
      
      So check if kptr_restrict is non zero and don't generate the syntethic
      PERF_RECORD_MMAP events for them.
      
      Warn the user about it in perf record and in perf report.
      
      In perf report the reference relocation symbol being zero means that
      kptr_restrict was set, thus /proc/kallsyms has only zeroed addresses, so don't
      use it to fixup symbol addresses when using a valid kallsyms (in the buildid
      cache) or vmlinux (in the vmlinux path) build-id located automatically or
      specified by the user.
      
      Provide an explanation about it in 'perf report' if kernel samples were taken,
      checking if a suitable vmlinux or kallsyms was found/specified.
      
      Restricted /proc/kallsyms don't go to the buildid cache anymore.
      
      Example:
      
       [acme@emilia ~]$ perf record -F 100000 sleep 1
      
       WARNING: Kernel address maps (/proc/{kallsyms,modules}) are restricted, check
       /proc/sys/kernel/kptr_restrict.
      
       Samples in kernel functions may not be resolved if a suitable vmlinux file is
       not found in the buildid cache or in the vmlinux path.
      
       Samples in kernel modules won't be resolved at all.
      
       If some relocation was applied (e.g. kexec) symbols may be misresolved even
       with a suitable vmlinux or kallsyms file.
      
       [ perf record: Woken up 1 times to write data ]
       [ perf record: Captured and wrote 0.005 MB perf.data (~231 samples) ]
       [acme@emilia ~]$
      
       [acme@emilia ~]$ perf report --stdio
       Kernel address maps (/proc/{kallsyms,modules}) were restricted,
       check /proc/sys/kernel/kptr_restrict before running 'perf record'.
      
       If some relocation was applied (e.g. kexec) symbols may be misresolved.
      
       Samples in kernel modules can't be resolved as well.
      
       # Events: 13  cycles
       #
       # Overhead  Command      Shared Object                 Symbol
       # ........  .......  .................  .....................
       #
          20.24%    sleep  [kernel.kallsyms]  [k] page_fault
          20.04%    sleep  [kernel.kallsyms]  [k] filemap_fault
          19.78%    sleep  [kernel.kallsyms]  [k] __lru_cache_add
          19.69%    sleep  ld-2.12.so         [.] memcpy
          14.71%    sleep  [kernel.kallsyms]  [k] dput
           4.70%    sleep  [kernel.kallsyms]  [k] flush_signal_handlers
           0.73%    sleep  [kernel.kallsyms]  [k] perf_event_comm
           0.11%    sleep  [kernel.kallsyms]  [k] native_write_msr_safe
      
       #
       # (For a higher level overview, try: perf report --sort comm,dso)
       #
       [acme@emilia ~]$
      
      This is because it found a suitable vmlinux (build-id checked) in
      /lib/modules/2.6.39-rc7+/build/vmlinux (use -v in perf report to see the long
      file name).
      
      If we remove that file from the vmlinux path:
      
       [root@emilia ~]# mv /lib/modules/2.6.39-rc7+/build/vmlinux \
      		     /lib/modules/2.6.39-rc7+/build/vmlinux.OFF
       [acme@emilia ~]$ perf report --stdio
       [kernel.kallsyms] with build id 57298cdbe0131f6871667ec0eaab4804dcf6f562
       not found, continuing without symbols
      
       Kernel address maps (/proc/{kallsyms,modules}) were restricted, check
       /proc/sys/kernel/kptr_restrict before running 'perf record'.
      
       As no suitable kallsyms nor vmlinux was found, kernel samples can't be
       resolved.
      
       Samples in kernel modules can't be resolved as well.
      
       # Events: 13  cycles
       #
       # Overhead  Command      Shared Object  Symbol
       # ........  .......  .................  ......
       #
          80.31%    sleep  [kernel.kallsyms]  [k] 0xffffffff8103425a
          19.69%    sleep  ld-2.12.so         [.] memcpy
      
       #
       # (For a higher level overview, try: perf report --sort comm,dso)
       #
       [acme@emilia ~]$
      Reported-by: NStephane Eranian <eranian@google.com>
      Suggested-by: NDavid Miller <davem@davemloft.net>
      Cc: Dave Jones <davej@redhat.com>
      Cc: David Miller <davem@davemloft.net>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Kees Cook <kees.cook@canonical.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Mike Galbraith <efault@gmx.de>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Pekka Enberg <penberg@cs.helsinki.fi>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Stephane Eranian <eranian@google.com>
      Cc: Tom Zanussi <tzanussi@gmail.com>
      Link: http://lkml.kernel.org/n/tip-mt512joaxxbhhp1odop04yit@git.kernel.orgSigned-off-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      ec80fde7
  2. 24 5月, 2011 1 次提交
  3. 23 5月, 2011 2 次提交
  4. 22 5月, 2011 6 次提交
  5. 21 5月, 2011 1 次提交
    • L
      sanitize <linux/prefetch.h> usage · 268bb0ce
      Linus Torvalds 提交于
      Commit e66eed65 ("list: remove prefetching from regular list
      iterators") removed the include of prefetch.h from list.h, which
      uncovered several cases that had apparently relied on that rather
      obscure header file dependency.
      
      So this fixes things up a bit, using
      
         grep -L linux/prefetch.h $(git grep -l '[^a-z_]prefetchw*(' -- '*.[ch]')
         grep -L 'prefetchw*(' $(git grep -l 'linux/prefetch.h' -- '*.[ch]')
      
      to guide us in finding files that either need <linux/prefetch.h>
      inclusion, or have it despite not needing it.
      
      There are more of them around (mostly network drivers), but this gets
      many core ones.
      Reported-by: NStephen Rothwell <sfr@canb.auug.org.au>
      Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>
      268bb0ce
  6. 19 5月, 2011 1 次提交
  7. 18 5月, 2011 1 次提交
    • S
      perf: Fix multi-event parsing bug · 94692349
      Stephane Eranian 提交于
      This patch fixes an issue with event parsing.
      The following commit appears to have broken the
      ability to specify a comma separated list of events:
      
         commit ceb53fbf
         Author: Ingo Molnar <mingo@elte.hu>
         Date:   Wed Apr 27 04:06:33 2011 +0200
      
             perf stat: Fail more clearly when an invalid modifier is specified
      
      This patch fixes this while preserving the desired effect:
      
      $ perf stat -e instructions:u,instructions:k ls /dev/null /dev/null
      
       Performance counter stats for 'ls /dev/null':
      
                  365956 instructions:u           #    0.00  insns per cycle
                  731806 instructions:k           #    0.00  insns per cycle
      
              0.001108862  seconds time elapsed
      
      $ perf stat -e task-clock-msecs true
      invalid event modifier: '-msecs'
      Run 'perf list' for a list of valid events and modifiers
      Signed-off-by: NStephane Eranian <eranian@google.com>
      Cc: acme@redhat.com
      Cc: peterz@infradead.org
      Cc: fweisbec@gmail.com
      Link: http://lkml.kernel.org/r/20110517133619.GA6999@quadSigned-off-by: NIngo Molnar <mingo@elte.hu>
      94692349
  8. 15 5月, 2011 2 次提交
    • A
      perf evlist: Fix per thread mmap setup · aece948f
      Arnaldo Carvalho de Melo 提交于
      The PERF_EVENT_IOC_SET_OUTPUT ioctl was returning -EINVAL when using
      --pid when monitoring multithreaded apps, as we can only share a ring
      buffer for events on the same thread if not doing per cpu.
      
      Fix it by using per thread ring buffers.
      
      Tested with:
      
      [root@felicio ~]# tuna -t 26131 -CP | nl
        1                      thread       ctxt_switches
        2    pid SCHED_ rtpri affinity voluntary nonvoluntary             cmd
        3 26131   OTHER     0      0,1  10814276      2397830 chromium-browse
        4  642    OTHER     0      0,1     14688            0 chromium-browse
        5  26148  OTHER     0      0,1    713602       115479 chromium-browse
        6  26149  OTHER     0      0,1    801958         2262 chromium-browse
        7  26150  OTHER     0      0,1   1271128          248 chromium-browse
        8  26151  OTHER     0      0,1         3            0 chromium-browse
        9  27049  OTHER     0      0,1     36796            9 chromium-browse
       10  618    OTHER     0      0,1     14711            0 chromium-browse
       11  661    OTHER     0      0,1     14593            0 chromium-browse
       12  29048  OTHER     0      0,1     28125            0 chromium-browse
       13  26143  OTHER     0      0,1   2202789          781 chromium-browse
      [root@felicio ~]#
      
      So 11 threads under pid 26131, then:
      
      [root@felicio ~]# perf record -F 50000 --pid 26131
      
      [root@felicio ~]# grep perf_event /proc/`pidof perf`/maps | nl
        1 7fa4a2538000-7fa4a25b9000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
        2 7fa4a25b9000-7fa4a263a000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
        3 7fa4a263a000-7fa4a26bb000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
        4 7fa4a26bb000-7fa4a273c000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
        5 7fa4a273c000-7fa4a27bd000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
        6 7fa4a27bd000-7fa4a283e000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
        7 7fa4a283e000-7fa4a28bf000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
        8 7fa4a28bf000-7fa4a2940000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
        9 7fa4a2940000-7fa4a29c1000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
       10 7fa4a29c1000-7fa4a2a42000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
       11 7fa4a2a42000-7fa4a2ac3000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
      [root@felicio ~]#
      
      11 mmaps, one per thread since we didn't specify any CPU list, so we need one
      mmap per thread and:
      
      [root@felicio ~]# perf record -F 50000 --pid 26131
      ^M
      ^C[ perf record: Woken up 79 times to write data ]
      [ perf record: Captured and wrote 20.614 MB perf.data (~900639 samples) ]
      
      [root@felicio ~]# perf report -D | grep PERF_RECORD_SAMPLE | cut -d/ -f2 | cut -d: -f1 | sort -n | uniq -c | sort -nr | nl
           1	 371310 26131
           2	  96516 26148
           3	  95694 26149
           4	  95203 26150
           5	   7291 26143
           6	     87 27049
           7	     76 661
           8	     60 29048
           9	     47 618
          10	     43 642
      [root@felicio ~]#
      
      Ok, one of the threads, 26151 was quiescent, so no samples there, but all the
      others are there.
      
      Then, if I specify one CPU:
      
      [root@felicio ~]# perf record -F 50000 --pid 26131 --cpu 1
      ^C[ perf record: Woken up 1 times to write data ]
      [ perf record: Captured and wrote 0.680 MB perf.data (~29730 samples) ]
      
      [root@felicio ~]# perf report -D | grep PERF_RECORD_SAMPLE | cut -d/ -f2 | cut -d: -f1 | sort -n | uniq -c | sort -nr | nl
           1	   8444 26131
           2	   2584 26149
           3	   2518 26148
           4	   2324 26150
           5	    123 26143
           6	      9 661
           7	      9 29048
      [root@felicio ~]#
      
      This machine has two cores, so fewer threads appeared on the radar, and:
      
      [root@felicio ~]# grep perf_event /proc/`pidof perf`/maps | nl
       1 7f484b922000-7f484b9a3000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
      [root@felicio ~]#
      
      Just one mmap, as now we can use just one per-cpu buffer instead of the
      per-thread needed in the previous case.
      
      For global profiling:
      
      [root@felicio ~]# perf record -F 50000 -a
      ^C[ perf record: Woken up 26 times to write data ]
      [ perf record: Captured and wrote 7.128 MB perf.data (~311412 samples) ]
      
      [root@felicio ~]# grep perf_event /proc/`pidof perf`/maps | nl
           1	7fb49b435000-7fb49b4b6000 rwxs 00000000 00:09 4064                       anon_inode:[perf_event]
           2	7fb49b4b6000-7fb49b537000 rwxs 00000000 00:09 4064                       anon_inode:[perf_event]
      [root@felicio ~]#
      
      It uses per-cpu buffers.
      
      For just one thread:
      
      [root@felicio ~]# perf record -F 50000 --tid 26148
      ^C[ perf record: Woken up 2 times to write data ]
      [ perf record: Captured and wrote 0.330 MB perf.data (~14426 samples) ]
      
      [root@felicio ~]# perf report -D | grep PERF_RECORD_SAMPLE | cut -d/ -f2 | cut -d: -f1 | sort -n | uniq -c | sort -nr | nl
           1	   9969 26148
      [root@felicio ~]#
      
      [root@felicio ~]# grep perf_event /proc/`pidof perf`/maps | nl
           1	7f286a51b000-7f286a59c000 rwxs 00000000 00:09 4064                       anon_inode:[perf_event]
      [root@felicio ~]#
      Tested-by: NDavid Ahern <dsahern@gmail.com>
      Tested-by: NLin Ming <ming.m.lin@intel.com>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Mike Galbraith <efault@gmx.de>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Stephane Eranian <eranian@google.com>
      Cc: Tom Zanussi <tzanussi@gmail.com>
      Link: http://lkml.kernel.org/r/20110426204401.GB1746@ghostprotocols.netSigned-off-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      aece948f
    • A
      perf tools: Honour the cpu list parameter when also monitoring a thread list · b9019418
      Arnaldo Carvalho de Melo 提交于
      The perf_evlist__create_maps was discarding the --cpu parameter when a
      --pid or --tid was specified, fix that.
      
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Mike Galbraith <efault@gmx.de>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Stephane Eranian <eranian@google.com>
      Cc: Tom Zanussi <tzanussi@gmail.com>
      Link: http://lkml.kernel.org/r/20110426204401.GB1746@ghostprotocols.netSigned-off-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      b9019418
  9. 10 5月, 2011 2 次提交
  10. 06 5月, 2011 1 次提交
  11. 30 4月, 2011 1 次提交
  12. 29 4月, 2011 2 次提交
  13. 27 4月, 2011 5 次提交
  14. 20 4月, 2011 1 次提交
  15. 19 4月, 2011 1 次提交
  16. 15 4月, 2011 2 次提交
  17. 08 4月, 2011 1 次提交
    • E
      perf: Fix a build error with some GCC versions · 621d2656
      Eric Dumazet 提交于
      Fix this:
      
       util/cgroup.c: In function ‘open_cgroup’:
       util/cgroup.c:16:16: error: ‘saved_ptr’ may be used uninitialized in this function
       util/cgroup.c:16:16: note: ‘saved_ptr’ was declared here
      
      Apparently newer GCC (4.6) can figure out that this variable is properly
      initialized - but some versions of GCC (such as 4.5.2) need help.
      Signed-off-by: NEric Dumazet <eric.dumazet@gmail.com>
      Cc: Arnaldo Carvalho de Melo <acme@infradead.org>
      LKML-Reference: <new-submission>
      Signed-off-by: NIngo Molnar <mingo@elte.hu>
      621d2656
  18. 06 4月, 2011 5 次提交
    • M
      perf probe: Fix listing incorrect line number with inline function · 1d46ea2a
      Masami Hiramatsu 提交于
      Fix a bug showing incorrect line number when a probe is put on the head of an
      inline function. This patch updates find_perf_probe_point() and introduces new
      rules to get correct line number.
      
       - If debuginfo doesn't have a correct file name, we shouldn't return line
         number too, because, without file name, line number is meaningless.
      
       - If the address is in a function, it stores the function name and the offset
         from the function entry.
      
         - If the address is on a line, it tries to get the relative line number from
           the function entry line, except for the address is same as the entry
           address of the function (in this case, the relative line number should
           be 0).
      
           - If the address is in an inline function entry (call-site), it uses the
             inline function call line number as the line on which the address is.
      
         - If the address is in an inline function body, it stores the inline
           function name and offset from the inline function call site instead of the
           (non-inlined) function.
      
      Cc: 2nddept-manager@sdl.hitachi.co.jp
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Lin Ming <ming.m.lin@intel.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      LKML-Reference: <20110330092605.2132.11629.stgit@ltc236.sdl.hitachi.co.jp>
      Signed-off-by: NMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Signed-off-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      1d46ea2a
    • M
      perf probe: Fix to find recursively inlined function · 1d878083
      Masami Hiramatsu 提交于
      Fix die_find_inlinefunc() to return correct innermost inlined function
      at given address. Without this fix, it returns the outermost inlined
      function.
      
      Cc: 2nddept-manager@sdl.hitachi.co.jp
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Lin Ming <ming.m.lin@intel.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      LKML-Reference: <20110330092559.2132.78634.stgit@ltc236.sdl.hitachi.co.jp>
      Signed-off-by: NMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Signed-off-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      1d878083
    • M
      perf probe: Fix multiple --vars options behavior · cc446446
      Masami Hiramatsu 提交于
      Fix a bug that perf-probe fails to initialize libdwfl and shows incorrect error
      when user gives multiple --vars options.
      
      Cc: 2nddept-manager@sdl.hitachi.co.jp
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Lin Ming <ming.m.lin@intel.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      LKML-Reference: <20110330092553.2132.42691.stgit@ltc236.sdl.hitachi.co.jp>
      Signed-off-by: NMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Signed-off-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      cc446446
    • M
      perf probe: Fix to remove redundant close · f0c4801a
      Masami Hiramatsu 提交于
      Since dwfl_end() closes given fd with dwfl, caller doesn't need to close its fd
      when finishing process.
      
      Cc: 2nddept-manager@sdl.hitachi.co.jp
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Lin Ming <ming.m.lin@intel.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      LKML-Reference: <20110330092547.2132.93728.stgit@ltc236.sdl.hitachi.co.jp>
      Signed-off-by: NMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Signed-off-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      f0c4801a
    • M
      perf probe: Fix to ensure function declared file · 7d21635a
      Masami Hiramatsu 提交于
      Fix to ensure function declared file matches given file name. This fixes
      a potential bug.
      
      As I've commented on Lin Ming's fastpath enhancement, decl_file should
      be checked on each probe point if user gives a probe point as func@file.
      
      Cc: 2nddept-manager@sdl.hitachi.co.jp
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Lin Ming <ming.m.lin@intel.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      LKML-Reference: <20110330092541.2132.3584.stgit@ltc236.sdl.hitachi.co.jp>
      Signed-off-by: NMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Signed-off-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      7d21635a
  19. 31 3月, 2011 1 次提交
  20. 30 3月, 2011 3 次提交
    • R
      perf tools: Fix NO_NEWT=1 python build error · 1b7155f7
      Robert Richter 提交于
      Fix the following build error:
      
           GEN python/perf.so
       In file included from util/evsel.h:10,
                        from util/python.c:6:
       util/hist.h:106:18: error: newt.h: No such file or directory
       error: command 'x86_64-pc-linux-gnu-gcc' failed with exit status 1
       make: *** [python/perf.so] Error 1
      
      by passing BASIC_CFLAGS to setup.py. BASIC_CFLAGS variable contains
      the -DNO_NEWT_SUPPORT switch which prevents building python c
      extension with newt.
      
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Stephane Eranian <eranian@google.com>
      LKML-Reference: <20110329180236.GA19366@erda.amd.com>
      Signed-off-by: NRobert Richter <robert.richter@amd.com>
      Signed-off-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      1b7155f7
    • D
      perf symbols: Properly align symbol_conf.priv_size · 4d439517
      David S. Miller 提交于
      If symbol_conf.priv_size is not a multiple of "sizeof(u64)" we'll bus
      error on sparc64 in symbol__new because the "struct symbol *" pointer
      is computed by adding symbol_conf.priv_size to the memory allocated.
      
      We cannot isolate the fix to symbol__new and symbol__delete since the
      private area is computed by subtracting the priv_size value from a
      "struct symbol" pointer, so then the private area can still be
      potentially unaligned.
      
      So, simply align the symbol_conf.priv_size value in symbol__init()
      
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      LKML-Reference: <20110328.175849.112593455.davem@davemloft.net>
      Signed-off-by: NDavid S. Miller <davem@davemloft.net>
      Signed-off-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      4d439517
    • L
      perf probe: Add fastpath to do lookup by function name · cd25f8bc
      Lin Ming 提交于
      v3 -> v2:
      - Make pubname_search_cb more generic
      - Add fastpath to find_probes also
      
      v2 -> v1:
      - Don't compare file names with cu_find_realpath(...), instead, compare
        them with the name returned by dwarf_decl_file(sp_die)
      
      The vmlinux file may have thousands of CUs.
      We can lookup function name from .debug_pubnames section
      to avoid the slow loop on CUs.
      
      1. Improvement data for find_line_range
      
      ./perf stat -e cycles -r 10 -- ./perf probe -k /home/mlin/vmlinux \
              -s /home/mlin/linux-2.6 \
              --line csum_partial_copy_to_user > tmp.log
      
      before patch applied
      =====================
             847,988,276 cycles
      
              0.355075856  seconds time elapsed
      
      after patch applied
      =====================
             206,102,622 cycles
      
              0.086883555  seconds time elapsed
      
      2. Improvement data for find_probes
      
      ./perf stat -e cycles -r 10 -- ./perf probe -k /home/mlin/vmlinux \
              -s /home/mlin/linux-2.6 \
              --vars csum_partial_copy_to_user > tmp.log
      
      before patch applied
      =====================
             848,490,844 cycles
      
              0.355307901  seconds time elapsed
      
      after patch applied
      =====================
             205,684,469 cycles
      
              0.086694010  seconds time elapsed
      Acked-by: NMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: linux-kernel <linux-kernel@vger.kernel.org>
      LKML-Reference: <1301041668.14111.52.camel@minggr.sh.intel.com>
      Signed-off-by: NLin Ming <ming.m.lin@intel.com>
      Signed-off-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      cd25f8bc