Skip to content
Snippets Groups Projects
  1. Aug 26, 2024
    • Steven Rostedt's avatar
      tracing/Documentation: Start a document on how to debug with tracing · 2fcd5aff
      Steven Rostedt authored
      Add a new document Documentation/trace/debugging.rst that will hold
      various ways to debug tracing.
      
      This initial version mentions trace_printk and how to create persistent
      buffers that can last across bootups.
      
      Cc: Masami Hiramatsu <mhiramat@kernel.org>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Vincent Donnefort <vdonnefort@google.com>
      Cc: Joel Fernandes <joel@joelfernandes.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Vineeth Pillai <vineeth@bitbyteword.org>
      Cc: Beau Belgrave <beaub@linux.microsoft.com>
      Cc: Alexander Graf <graf@amazon.com>
      Cc: Baoquan He <bhe@redhat.com>
      Cc: Borislav Petkov <bp@alien8.de>
      Cc: "Paul E. McKenney" <paulmck@kernel.org>
      Cc: David Howells <dhowells@redhat.com>
      Cc: Mike Rapoport <rppt@kernel.org>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Tony Luck <tony.luck@intel.com>
      Cc: Guenter Roeck <linux@roeck-us.net>
      Cc: Ross Zwisler <zwisler@google.com>
      Cc: Kees Cook <keescook@chromium.org>
      Cc: Alexander Aring <aahringo@redhat.com>
      Cc: "Luis Claudio R. Goncalves" <lgoncalv@redhat.com>
      Cc: Tomas Glozar <tglozar@redhat.com>
      Cc: John Kacur <jkacur@redhat.com>
      Cc: Clark Williams <williams@redhat.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: "Jonathan Corbet" <corbet@lwn.net>
      Link: https://lore.kernel.org/20240823014019.702433486@goodmis.org
      
      
      Signed-off-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
      2fcd5aff
    • Steven Rostedt's avatar
      tracing: Add option to set an instance to be the trace_printk destination · ef2bd81d
      Steven Rostedt authored
      Add a option "trace_printk_dest" that will make the tracing instance the
      location that trace_printk() will go to. This is useful if the
      trace_printk or one of the top level tracers is too noisy and there's a
      need to separate the two. Then an instance can be created, the
      trace_printk can be set to go there instead, where it will not be lost in
      the noise of the top level tracer.
      
      Note, only one instance can be the destination of trace_printk at a time.
      If an instance sets this flag, the instance that had it set will have it
      cleared. There is always one instance that has this set. By default, that
      is the top instance. This flag cannot be cleared from the top instance.
      Doing so will result in an -EINVAL. The only way this flag can be cleared
      from the top instance is by another instance setting it.
      
      Cc: Masami Hiramatsu <mhiramat@kernel.org>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Vincent Donnefort <vdonnefort@google.com>
      Cc: Joel Fernandes <joel@joelfernandes.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Vineeth Pillai <vineeth@bitbyteword.org>
      Cc: Beau Belgrave <beaub@linux.microsoft.com>
      Cc: Alexander Graf <graf@amazon.com>
      Cc: Baoquan He <bhe@redhat.com>
      Cc: Borislav Petkov <bp@alien8.de>
      Cc: "Paul E. McKenney" <paulmck@kernel.org>
      Cc: David Howells <dhowells@redhat.com>
      Cc: Mike Rapoport <rppt@kernel.org>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Tony Luck <tony.luck@intel.com>
      Cc: Guenter Roeck <linux@roeck-us.net>
      Cc: Ross Zwisler <zwisler@google.com>
      Cc: Kees Cook <keescook@chromium.org>
      Cc: Alexander Aring <aahringo@redhat.com>
      Cc: "Luis Claudio R. Goncalves" <lgoncalv@redhat.com>
      Cc: Tomas Glozar <tglozar@redhat.com>
      Cc: John Kacur <jkacur@redhat.com>
      Cc: Clark Williams <williams@redhat.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: "Jonathan Corbet" <corbet@lwn.net>
      Link: https://lore.kernel.org/20240823014019.545459018@goodmis.org
      
      
      Signed-off-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
      ef2bd81d
  2. Jul 01, 2024
  3. Jun 11, 2024
    • Steven Rostedt (Google)'s avatar
      function_graph: Everyone uses HAVE_FUNCTION_GRAPH_RET_ADDR_PTR, remove it · 5f7fb89a
      Steven Rostedt (Google) authored
      All architectures that implement function graph also implements
      HAVE_FUNCTION_GRAPH_RET_ADDR_PTR. Remove it, as it is no longer a
      differentiator.
      
      Link: https://lore.kernel.org/linux-trace-kernel/20240611031737.982047614@goodmis.org
      
      
      
      Cc: Masami Hiramatsu <mhiramat@kernel.org>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Will Deacon <will@kernel.org>
      Cc: Guo Ren <guoren@kernel.org>
      Cc: Huacai Chen <chenhuacai@kernel.org>
      Cc: WANG Xuerui <kernel@xen0n.name>
      Cc: Michael Ellerman <mpe@ellerman.id.au>
      Cc: Nicholas Piggin <npiggin@gmail.com>
      Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
      Cc: "Naveen N. Rao" <naveen.n.rao@linux.ibm.com>
      Cc: Paul Walmsley <paul.walmsley@sifive.com>
      Cc: Palmer Dabbelt <palmer@dabbelt.com>
      Cc: Albert Ou <aou@eecs.berkeley.edu>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: Borislav Petkov <bp@alien8.de>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Signed-off-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
      5f7fb89a
  4. May 13, 2024
  5. May 07, 2024
  6. May 02, 2024
  7. May 01, 2024
  8. Apr 02, 2024
  9. Mar 18, 2024
  10. Mar 06, 2024
  11. Jan 11, 2024
  12. Jan 03, 2024
  13. Dec 21, 2023
  14. Nov 17, 2023
  15. Nov 16, 2023
  16. Nov 10, 2023
  17. Oct 17, 2023
  18. Oct 04, 2023
  19. Sep 11, 2023
  20. Aug 23, 2023
  21. Aug 22, 2023
  22. Aug 18, 2023
  23. Jun 23, 2023
  24. Jun 22, 2023
    • Daniel Bristot de Oliveira's avatar
      tracing/timerlat: Add user-space interface · e88ed227
      Daniel Bristot de Oliveira authored
      Going a step further, we propose a way to use any user-space
      workload as the task waiting for the timerlat timer. This is done
      via a per-CPU file named osnoise/cpu$id/timerlat_fd file.
      
      The tracef_fd allows a task to open at a time. When a task reads
      the file, the timerlat timer is armed for future osnoise/timerlat_period_us
      time. When the timer fires, it prints the IRQ latency and
      wakes up the user-space thread waiting in the timerlat_fd.
      
      The thread then starts to run, executes the timerlat measurement, prints
      the thread scheduling latency and returns to user-space.
      
      When the thread rereads the timerlat_fd, the tracer will print the
      user-ret(urn) latency, which is an additional metric.
      
      This additional metric is also traced by the tracer and can be used, for
      example of measuring the context switch overhead from kernel-to-user and
      user-to-kernel, or the response time for an arbitrary execution in
      user-space.
      
      The tracer supports one thread per CPU, the thread must be pinned to
      the CPU, and it cannot migrate while holding the timerlat_fd. The reason
      is that the tracer is per CPU (nothing prohibits the tracer from
      allowing migrations in the future). The tracer monitors the migration
      of the thread and disables the tracer if detected.
      
      The timerlat_fd is only available for opening/reading when timerlat
      tracer is enabled, and NO_OSNOISE_WORKLOAD is set.
      
      The simplest way to activate this feature from user-space is:
      
       -------------------------------- %< -----------------------------------
       int main(void)
       {
      	char buffer[1024];
      	int timerlat_fd;
      	int retval;
      	long cpu = 0;	/* place in CPU 0 */
      	cpu_set_t set;
      
      	CPU_ZERO(&set);
      	CPU_SET(cpu, &set);
      
      	if (sched_setaffinity(gettid(), sizeof(set), &set) == -1)
      		return 1;
      
      	snprintf(buffer, sizeof(buffer),
      		"/sys/kernel/tracing/osnoise/per_cpu/cpu%ld/timerlat_fd",
      		cpu);
      
      	timerlat_fd = open(buffer, O_RDONLY);
      	if (timerlat_fd < 0) {
      		printf("error opening %s: %s\n", buffer, strerror(errno));
      		exit(1);
      	}
      
      	for (;;) {
      		retval = read(timerlat_fd, buffer, 1024);
      		if (retval < 0)
      			break;
      	}
      
      	close(timerlat_fd);
      	exit(0);
      }
       -------------------------------- >% -----------------------------------
      
      When disabling timerlat, if there is a workload holding the timerlat_fd,
      the SIGKILL will be sent to the thread.
      
      Link: https://lkml.kernel.org/r/69fe66a863d2792ff4c3a149bf9e32e26468bb3a.1686063934.git.bristot@kernel.org
      
      
      
      Cc: Juri Lelli <juri.lelli@redhat.com>
      Cc: William White <chwhite@redhat.com>
      Cc: Daniel Bristot de Oliveira <bristot@kernel.org>
      Cc: Masami Hiramatsu <mhiramat@kernel.org>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Signed-off-by: default avatarDaniel Bristot de Oliveira <bristot@kernel.org>
      Signed-off-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
      e88ed227
    • Jiri Olsa's avatar
      ftrace: Show all functions with addresses in available_filter_functions_addrs · 83f74441
      Jiri Olsa authored
      Adding new available_filter_functions_addrs file that shows all available
      functions (same as available_filter_functions) together with addresses,
      like:
      
        # cat available_filter_functions_addrs | head
        ffffffff81000770 __traceiter_initcall_level
        ffffffff810007c0 __traceiter_initcall_start
        ffffffff81000810 __traceiter_initcall_finish
        ffffffff81000860 trace_initcall_finish_cb
        ...
      
      Note displayed address is the patch-site address and can differ from
      /proc/kallsyms address.
      
      It's useful to have address avilable for traceable symbols, so we don't
      need to allways cross check kallsyms with available_filter_functions
      (or the other way around) and have all the data in single file.
      
      For backwards compatibility reasons we can't change the existing
      available_filter_functions file output, but we need to add new file.
      
      The problem is that we need to do 2 passes:
      
       - through available_filter_functions and find out if the function is traceable
       - through /proc/kallsyms to get the address for traceable function
      
      Having available_filter_functions symbols together with addresses allow
      us to skip the kallsyms step and we are ok with the address in
      available_filter_functions_addr not being the function entry, because
      kprobe_multi uses fprobe and that handles both entry and patch-site
      address properly.
      
      We have 2 interfaces how to create kprobe_multi link:
      
        a) passing symbols to kernel
      
           1) user gathers symbols and need to ensure that they are
              trace-able -> pass through available_filter_functions file
      
           2) kernel takes those symbols and translates them to addresses
              through kallsyms api
      
           3) addresses are passed to fprobe/ftrace through:
      
               register_fprobe_ips
               -> ftrace_set_filter_ips
      
        b) passing addresses to kernel
      
           1) user gathers symbols and needs to ensure that they are
              trace-able -> pass through available_filter_functions file
      
           2) user takes those symbols and translates them to addresses
             through /proc/kallsyms
      
           3) addresses are passed to the kernel and kernel calls:
      
               register_fprobe_ips
               -> ftrace_set_filter_ips
      
      The new available_filter_functions_addrs file helps us with option b),
      because we can make 'b 1' and 'b 2' in one step - while filtering traceable
      functions, we get the address directly.
      
      Link: https://lore.kernel.org/linux-trace-kernel/20230611130029.1202298-1-jolsa@kernel.org
      
      
      
      Cc: Masami Hiramatsu <mhiramat@kernel.org>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
      Tested-by: Jackie Liu <liuyun01@kylinos.cn> # x86
      Suggested-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
      Suggested-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarJiri Olsa <jolsa@kernel.org>
      Signed-off-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
      83f74441
  25. Jun 21, 2023
  26. Jun 20, 2023
  27. Jun 15, 2023
  28. Jun 14, 2023
  29. Jun 06, 2023
Loading