This change enables the db export api to export callchains. This is
accomplished by adding callchains obtained from samples to the
call_path_root structure and exporting them via the current call path
export API.
While the current API does support exporting call paths, this is not
supported when sampling. This commit addresses that missing feature by
allowing the export of call paths when callchains are present in
samples.
Summary:
- This feature is activated by initializing the call_path_root member
inside the db_export structure to a non-null value.
- Callchains are resolved with thread__resolve_callchain() and then stored
and exported by adding a call path under call path root.
- Symbol and DSO for each callchain node are exported via db_ids_from_al()
This commit puts in place infrastructure to be used by subsequent commits,
and by itself, does not introduce any user-visible changes.
Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461831551-12213-4-git-send-email-cphlipot0@gmail.com
[ Made adjustments suggested by Adrian Hunter, see thread via this cset's Link: tag ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Move the call path handling code out of thread-stack.c and
thread-stack.h to allow other components that are not part of
thread-stack to create call paths.
Summary:
- Create call-path.c and call-path.h and add them to the build.
- Move all call path related code out of thread-stack.c and thread-stack.h
and into call-path.c and call-path.h.
- A small subset of structures and functions are now visible through
call-path.h, which is required for thread-stack.c to continue to
compile.
This change is a prerequisite for subsequent patches in this change set
and by itself contains no user-visible changes.
Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461831551-12213-3-git-send-email-cphlipot0@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The existing implementation of thread__resolve_callchain, under certain
circumstances, can assemble callchain entries in the incorrect order.
The callchain entries are resolved incorrectly for a sample when all of
the following conditions are met:
1. callchain_param.order is set to ORDER_CALLER
2. thread__resolve_callchain_sample is able to resolve callchain entries
for the sample.
3. unwind__get_entries is also able to resolve callchain entries for the
sample.
The fix is accomplished by reversing the order in which
thread__resolve_callchain_sample and unwind__get_entries are called when
callchain_param.order is set to ORDER_CALLER.
Unwind specific code from thread__resolve_callchain is also moved into a
new static function to improve readability of the fix.
How to Reproduce the Existing Bug:
Modifying perf script to print call trees in the opposite order or
applying the remaining patches from this series and comparing the
results output from export-to-postgtresql.py are the easiest ways to see
the bug, however it can still be seen in current builds using perf
report.
Here is how i can reproduce the bug using perf report:
# perf record --call-graph=dwarf stress -c 1 -t 5
when i run this command:
# perf report --call-graph=flat,0,0,callee
This callchain, containing kernel (handle_irq_event, etc) and userspace
samples (__libc_start_main, etc) is contained in the output, which looks
correct (callee order):
gen8_irq_handler
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
ret_from_intr
__random
rand
0x558f2a04dded
0x558f2a04c774
__libc_start_main
0x558f2a04dcd9
Now run this command using caller order:
# perf report --call-graph=flat,0,0,caller
It is expected to see the exact reverse of the above when using caller
order (with "0x558f2a04dcd9" at the top and "gen8_irq_handler" at the
bottom) in the output, but it is nowhere to be found.
instead you see this:
ret_from_intr
do_IRQ
handle_irq
handle_edge_irq
handle_irq_event
handle_irq_event_percpu
gen8_irq_handler
0x558f2a04dcd9
__libc_start_main
0x558f2a04c774
0x558f2a04dded
rand
__random
Notice how internally the kernel symbols are reversed and the user space
symbols are reversed, but the kernel symbols still appear above the user
space symbols.
if this patch is applied and perf script is re-run, you will see the
expected output (with "0x558f2a04dcd9" at the top and "gen8_irq_handler"
at the bottom):
0x558f2a04dcd9
__libc_start_main
0x558f2a04c774
0x558f2a04dded
rand
__random
ret_from_intr
do_IRQ
handle_irq
handle_edge_irq
handle_irq_event
handle_irq_event_percpu
gen8_irq_handler
Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461831551-12213-2-git-send-email-cphlipot0@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In perf_mmap__read(), give better names to pointers. Original name 'old'
and 'head' directly related to pointers in ring buffer control page. For
backward ring buffer, the meaning of 'head' point is not 'the first byte
of free space', but 'the first byte of the last record'. To reduce
confusion, rename 'old' to 'start', 'head' to 'end'. 'start' -> 'end'
is the direction the records should be read from.
Change parameter order.
Change 'overwrite' to 'check_messup'. When reading from 'head', no need
to check messup for for backward ring buffer.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1461723563-67451-3-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
ppc64le functions have a Global Entry Point (GEP) and a Local Entry
Point (LEP). While placing a probe, we always prefer the LEP since it
catches function calls through both the GEP and the LEP. In order to do
this, we fixup the function entry points during elf symbol table lookup
to point to the LEPs. This works, but breaks 'perf test kallsyms' since
the symbols loaded from the symbol table (pointing to the LEP) do not
match the symbols in kallsyms.
To fix this, we do not adjust all the symbols during symbol table load.
Instead, we note down st_other in a newly introduced arch-specific
member of perf symbol structure, and later use this to adjust the probe
trace point.
Reported-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Acked-by: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Acked-by: Balbir Singh <bsingharora@gmail.com>
Cc: Mark Wielaard <mjw@redhat.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Thiago Jung Bauermann <bauerman@linux.vnet.ibm.com>
Cc: linuxppc-dev@lists.ozlabs.org
Link: http://lkml.kernel.org/r/6be7c2b17e370100c2f79dd444509df7929bdd3e.1460451721.git.naveen.n.rao@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Sometimes we want to sort an existing rbtree by a different key,
introduce a template for that, that needs only to be provided the
rbtree root and the number of entries in it.
To do that a new rbtree will be created with extra space for each entry,
where possibly pre-calculated keys will be stored to be used in the
resort process and also later, when using the newly sorted rbtree.
Please check the following two changesets to see it in use for resorting
stats for threads and its syscalls in 'perf trace --summary'.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-9l6e1q34lmf3wwdeewstyakg@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When the kernel allows tweaking perf_event_max_stack and the event being
setup has PERF_SAMPLE_CALLCHAIN in its perf_event_attr.sample_type, tell
the user that tweaking /proc/sys/kernel/perf_event_max_stack may solve
the problem.
Before:
# echo 32000 > /proc/sys/kernel/perf_event_max_stack
# perf record -g usleep 1
Error:
The sys_perf_event_open() syscall returned with 12 (Cannot allocate memory) for event (cycles:ppp).
/bin/dmesg may provide additional information.
No CONFIG_PERF_EVENTS=y kernel support configured?
#
After:
# echo 64000 > /proc/sys/kernel/perf_event_max_stack
# perf record -g usleep 1
Error:
Not enough memory to setup event with callchain.
Hint: Try tweaking /proc/sys/kernel/perf_event_max_stack
Hint: Current value: 64000
#
Suggested-by: David Ahern <dsahern@gmail.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-ebv0orelj1s1ye857vhb82ov@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Perf is not able to register probe in kernel module when dwarf supprt
is not there(and so it goes for symtab). Perf passes full path of
module where only module name is required which is causing the problem.
This patch fixes this issue.
Before applying patch:
$ dpkg -s libdw-dev
dpkg-query: package 'libdw-dev' is not installed and no information is...
$ sudo ./perf probe -m /linux/samples/kprobes/kprobe_example.ko kprobe_init
Added new event:
probe:kprobe_init (on kprobe_init in /linux/samples/kprobes/kprobe_example.ko)
You can now use it in all perf tools, such as:
perf record -e probe:kprobe_init -aR sleep 1
$ sudo cat /sys/kernel/debug/tracing/kprobe_events
p:probe/kprobe_init /linux/samples/kprobes/kprobe_example.ko:kprobe_init
$ sudo ./perf record -a -e probe:kprobe_init
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.105 MB perf.data ]
$ sudo ./perf script # No output here
After applying patch:
$ sudo ./perf probe -m /linux/samples/kprobes/kprobe_example.ko kprobe_init
Added new event:
probe:kprobe_init (on kprobe_init in kprobe_example)
You can now use it in all perf tools, such as:
perf record -e probe:kprobe_init -aR sleep 1
$ sudo cat /sys/kernel/debug/tracing/kprobe_events
p:probe/kprobe_init kprobe_example:kprobe_init
$ sudo ./perf record -a -e probe:kprobe_init
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.105 MB perf.data (2 samples) ]
$ sudo ./perf script
insmod 13990 [002] 5961.216833: probe:kprobe_init: ...
insmod 13995 [002] 5962.889384: probe:kprobe_init: ...
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1461680741-12517-1-git-send-email-ravi.bangoria@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Perf can add a probe on kernel module which has not been loaded yet.
The current implementation finds the module name from path. But if the
filename is different from the actual module name then perf fails to
register a probe while loading module because of mismatch in the names.
For example, samples/kobject/kobject-example.ko is loaded as
kobject_example.
Before applying patch:
$ sudo ./perf probe -m /linux/samples/kobject/kobject-example.ko foo_show
Added new event:
probe:foo_show (on foo_show in kobject-example)
You can now use it in all perf tools, such as:
perf record -e probe:foo_show -aR sleep 1
$ cat /sys/kernel/debug/tracing/kprobe_events
p:probe/foo_show kobject-example:foo_show
$ insmod kobject-example.ko
$ lsmod
Module Size Used by
kobject_example 16384 0
Generate read to /sys/kernel/kobject_example/foo while recording data
with below command
$ sudo ./perf record -e probe:foo_show -a
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.093 MB perf.data ]
$./perf report --stdio -F overhead,comm,dso,sym
Error:
The perf.data.old file has no samples!
After applying patch:
$ sudo ./perf probe -m /linux/samples/kobject/kobject-example.ko foo_show
Added new event:
probe:foo_show (on foo_show in kobject_example)
You can now use it in all perf tools, such as:
perf record -e probe:foo_show -aR sleep 1
$ sudo cat /sys/kernel/debug/tracing/kprobe_events
p:probe/foo_show kobject_example:foo_show
$ insmod kobject-example.ko
$ lsmod
Module Size Used by
kobject_example 16384 0
Generate read to /sys/kernel/kobject_example/foo while recording data
with below command
$ sudo ./perf record -e probe:foo_show -a
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.097 MB perf.data (8 samples) ]
$ sudo ./perf report --stdio -F overhead,comm,dso,sym
...
# Samples: 8 of event 'probe:foo_show'
# Event count (approx.): 8
#
# Overhead Command Shared Object Symbol
# ........ ....... ................. ............
#
100.00% cat [kobject_example] [k] foo_show
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1461680741-12517-2-git-send-email-ravi.bangoria@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Don't read broken data after 'head' pointer.
Following commits will feed perf_evlist__mmap_read() with some 'head'
pointers not maintained by kernel. If 'head' pointer breaks an event, we
should avoid reading from the broken event. This can happen in backward
ring buffer.
For example:
old head
| |
V V
+---+------+----------+----+-----+--+
|..E|D....D|C........C|B..B|A....|E.|
+---+------+----------+----+-----+--+
'old' pointer points to the beginning of 'A' and trying read from it,
but 'A' has been overwritten. In this case, don't try to read from 'A',
simply return NULL.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1461637738-62722-2-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
write_buildid() increments 'name_len' with intention to take into
account trailing zero byte. However, 'name_len' was already incremented
in machine__write_buildid_table() before. So this leads to
out-of-bounds read in do_write():
$ ./perf record sleep 0
[ perf record: Woken up 1 times to write data ]
=================================================================
==15899==ERROR: AddressSanitizer: global-buffer-overflow on address 0x00000099fc92 at pc 0x7f1aa9c7eab5 bp 0x7fff940f84d0 sp 0x7fff940f7c78
READ of size 19 at 0x00000099fc92 thread T0
#0 0x7f1aa9c7eab4 (/usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/libasan.so.2+0x44ab4)
#1 0x649c5b in do_write util/header.c:67
#2 0x649c5b in write_padded util/header.c:82
#3 0x57e8bc in write_buildid util/build-id.c:239
#4 0x57e8bc in machine__write_buildid_table util/build-id.c:278
...
0x00000099fc92 is located 0 bytes to the right of global variable '*.LC99' defined in 'util/symbol.c' (0x99fc80) of size 18
'*.LC99' is ascii string '[kernel.kallsyms]'
...
Shadow bytes around the buggy address:
0x00008012bf80: f9 f9 f9 f9 00 00 00 00 00 00 03 f9 f9 f9 f9 f9
=>0x00008012bf90: 00 00[02]f9 f9 f9 f9 f9 00 00 00 00 00 05 f9 f9
0x00008012bfa0: f9 f9 f9 f9 00 03 f9 f9 f9 f9 f9 f9 00 00 00 00
Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461053847-5633-1-git-send-email-aryabinin@virtuozzo.com
[ Remove the off-by one at the origin, to keep len(s) == strlen(s) assumption ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Pull perf/core improvements and fixes from Arnaldo Carvalho de Melo:
Build fixes:
- Fix 'perf trace' build when DWARF unwind isn't available (Arnaldo Carvalho de Melo)
- Remove x86 references from arch-neutral Build, fixing it in !x86 arches,
reported as breaking the build for powerpc64le in linux-next (Arnaldo Carvalho de Melo)
Infrastructure changes:
- Do memset() variable 'st' using the correct size in the jit code (Colin Ian King)
- Fix postgresql ubuntu 'perf script' install instructions (Chris Phlipot)
- Use callchain_param more thoroughly when checking how callchains were
configured, eventually will be the only way to look for callchain parameters
(Arnaldo Carvalho de Melo)
- Fix some issues in the 'perf test kallsyms' entry (Arnaldo Carvalho de Melo)
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
The current code is memsetting the 'struct stat' variable 'st' with the size of
'stat' (which turns out to be 1 byte) rather than the size of variable 'sz'.
Committer notes:
sizeof(function) isn't valid, the result depends on the compiler used, with
gcc, enabling pedantic warnings we get:
$ cat sizeof_function.c
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <stdio.h>
int main(void)
{
printf("sizeof(stat)=%zd, stat=%p\n", sizeof(stat), stat);
return 0;
}
$ readelf -sW sizeof_function | grep -w stat
49: 0000000000400630 16 FUNC WEAK HIDDEN 13 stat
$ cc -pedantic sizeof_function.c -o sizeof_function
sizeof_function.c: In function ‘main’:
sizeof_function.c:8:46: warning: invalid application of ‘sizeof’ to a function type [-Wpointer-arith]
printf("sizeof(stat)=%zd, stat=%p\n", sizeof(stat), stat);
^
$ ./sizeof_function
sizeof(stat)=1, stat=0x400630
$
Standard C, section 6.5.3.4:
"The sizeof operator shall not be applied to an expression that has function
type or an incomplete type, to the parenthesized name of such a type,
or to an expression that designates a bit-field member."
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf
Signed-off-by: Colin Ian King <colin.king@canonical.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Fixes: 9b07e27f88 ("perf inject: Add jitdump mmap injection support")
Link: http://lkml.kernel.org/r/1461020838-9260-1-git-send-email-colin.king@canonical.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Tracing a workload that uses transactions gave a seg fault as follows:
perf record -e intel_pt// workload
perf report
Program received signal SIGSEGV, Segmentation fault.
0x000000000054b58c in intel_pt_reset_last_branch_rb (ptq=0x1a36110)
at util/intel-pt.c:929
929 ptq->last_branch_rb->nr = 0;
(gdb) p ptq->last_branch_rb
$1 = (struct branch_stack *) 0x0
(gdb) up
1148 intel_pt_reset_last_branch_rb(ptq);
(gdb) l
1143 if (ret)
1144 pr_err("Intel Processor Trace: failed to deliver transaction event
1145 ret);
1146
1147 if (pt->synth_opts.callchain)
1148 intel_pt_reset_last_branch_rb(ptq);
1149
1150 return ret;
1151 }
1152
(gdb) p pt->synth_opts.callchain
$2 = true
(gdb)
(gdb) bt
#0 0x000000000054b58c in intel_pt_reset_last_branch_rb (ptq=0x1a36110)
#1 0x000000000054c1e0 in intel_pt_synth_transaction_sample (ptq=0x1a36110)
#2 0x000000000054c5b2 in intel_pt_sample (ptq=0x1a36110)
Caused by checking the 'callchain' flag when it should have been the
'last_branch' flag. Fix that.
Reported-by: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: stable@vger.kernel.org # v4.4+
Fixes: f14445ee72 ("perf intel-pt: Support generating branch stack")
Link: http://lkml.kernel.org/r/1460977068-11566-1-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
# perf test -v python
16: Try 'import perf' in python, checking link problems :
--- start ---
test child forked, pid 672
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /tmp/build/perf/python/perf.so: undefined symbol:
symbol_conf
test child finished with -1
---- end ----
Try 'import perf' in python, checking link problems: FAILED!
#
To fix it just pass a parameter to perf_evsel__fprintf_sym telling if
callchains should be printed.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-comrsr20bsnr8bg0n6rfwv12@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
