Skip to content

Update diagnostics docs for dotnet-trace collect-linux #52273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mdh1418 wants to merge 1 commit into
base: main
Choose a base branch
from
+34 −8
Open

Update diagnostics docs for dotnet-trace collect-linux #52273

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/core/diagnostics/debug-highcpu.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ At this point, you can safely say the CPU is running higher than you expect. Ide

## Analyze High CPU with Profiler

When analyzing an app with high CPU usage, you need a diagnostics tool that can provide insights into what the code is doing. The usual choice is a profiler, and there are different profiler options to choose from. `dotnet-trace` can be used on all operating systems, however, its limitations of safe-point bias and managed-only callstacks result in more general information compared to a kernel-aware profiler like 'perf' for Linux or ETW for Windows. If your performance investigation involves only managed code, generally `dotnet-trace` will be sufficient.
When analyzing an app with high CPU usage, use a profiler to understand what the code is doing. `dotnet-trace` works on all operating systems, but safe-point bias and managed-only callstacks limit it to more general information than a kernel-aware profiler like ETW for Windows or `perf` for Linux. On Linux, [`dotnet-trace collect-linux`](dotnet-trace.md#dotnet-trace-collect-linux) eliminates these limitations by combining EventPipe with OS-level perf_events in a single unified trace. If your performance investigation involves only managed code, `dotnet-trace collect` is generally sufficient.

Copy link
Member

@brianrob brianrob Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say that dotnet-trace collect is often sufficient for managed code investigations within a single process. But often not knowing what native runtime frames are on the stack makes the investigations difficult. I would recommend dotnet-trace collect-linux on Linux for any .NET 10+ investigations. The set of tools is the same, so it seems ideal to bias towards a higher fidelity trace.

Copy link
Member

@noahfalk noahfalk Mar 12, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fully agree on the recommendation. In terms of how we phrase it in this doc I'd suggest we keep the initial blurb above very simple, just mentioning that 'dotnet-trace collect' is a cross-platform option and that depending on OS and .NET version improved capabilities may be available. Then in the tabs below we can give more refined guidance that is Windows or Linux specific.

In order to make the doc self-consistent if we mention dotnet-trace collect-linux as a recommended option then we need to provide a guide that tells people how to use it. Right now the section below only tells users about 'perf'.

Copy link
Member

@brianrob brianrob Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with your feedback here @noahfalk.

### [Linux](#tab/linux)

Expand Down
24 changes: 24 additions & 0 deletions docs/core/diagnostics/dotnet-trace.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -699,6 +699,30 @@ However, when you want to gain a finer control over the lifetime of the app bein

## (Linux-only) Collect a machine-wide trace using dotnet-trace

### Get symbols for native runtime frames
Copy link
Member

@noahfalk noahfalk Mar 12, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought Beau told us earlier that acquiring debug binaries was unnecessary and has no bearing on the collect-linux stackwalking results? Did that prove to be untrue?

Copy link
Member Author

@mdh1418 mdh1418 Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll need to investigate that, I might still be hitting the non-deterministic symbol resolution issue.


`collect-linux` captures native frames in callstacks. To resolve native method names for runtime libraries (such as `libcoreclr.so`), place the corresponding debug symbol files on disk beside the libraries. Without these symbols, native frames appear as unresolved addresses in the trace.

Unlike [`perfcollect`](./trace-perfcollect-lttng.md), `collect-linux` doesn't require you to set environment variables like `DOTNET_PerfMapEnabled` or `DOTNET_EnableEventLog` before starting your application. `collect-linux` dynamically enables perfmap generation for JIT-compiled code when the trace begins, so you don't need to restart any .NET processes.
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"perfmap" should be "perf map" (or "perf maps") to match the terminology used elsewhere (for example, "Export perf maps and jit dumps").

Copilot uses AI. Check for mistakes.

To download native runtime symbols, use [dotnet-symbol](./dotnet-symbol.md):

1. Install `dotnet-symbol`:

```bash
dotnet tool install -g dotnet-symbol
```

1. Download the debug symbols for your runtime version. For example, if your runtime is installed at `/usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.0`:

```bash
Comment on lines +712 to +718
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These code blocks run dotnet commands; this article generally uses dotnetcli fenced blocks for CLI examples. Consider switching the fence info strings from bash to dotnetcli here for consistency and proper styling.

Suggested change
```bash
dotnet tool install -g dotnet-symbol
```
1. Download the debug symbols for your runtime version. For example, if your runtime is installed at `/usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.0`:
```bash
```dotnetcli
dotnet tool install -g dotnet-symbol

  1. Download the debug symbols for your runtime version. For example, if your runtime is installed at /usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.0:

Copilot uses AI. Check for mistakes.
dotnet-symbol --symbols /usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.0/lib*.so
```

1. Place the downloaded `.so.dbg` files beside the runtime libraries they correspond to (for example, `libcoreclr.so.dbg` next to `libcoreclr.so`). If you run `dotnet-symbol` from the runtime directory, it places the symbols there automatically.
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sentence about needing to run dotnet-symbol from the runtime directory is misleading. By default, dotnet-symbol writes next to the input file (or you can use -o/--output), and placing .so.dbg files under /usr/share/dotnet/... typically requires elevated permissions. Consider rewording to describe the default output behavior and mention using sudo or --output + copy if needed.

Suggested change
1. Place the downloaded `.so.dbg` files beside the runtime libraries they correspond to (for example, `libcoreclr.so.dbg` next to `libcoreclr.so`). If you run `dotnet-symbol` from the runtime directory, it places the symbols there automatically.
1. Place the downloaded `.so.dbg` files beside the runtime libraries they correspond to (for example, `libcoreclr.so.dbg` next to `libcoreclr.so`). By default, `dotnet-symbol` writes symbol files next to each input file. If your runtime libraries live under a protected path such as `/usr/share/dotnet/...`, run `dotnet-symbol` with elevated permissions (for example, by using `sudo`), or use the `-o`/`--output` option to write to a writable directory, then copy the `.so.dbg` files beside the runtime libraries.

Copilot uses AI. Check for mistakes.

After you place the symbols, `collect-linux` resolves native method names when it processes the trace.
Copy link
Member

@brianrob brianrob Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
After you place the symbols, `collect-linux` resolves native method names when it processes the trace.
After you place the symbols, `collect-linux` resolves native method names when it collects the trace.


This example captures CPU samples for all processes on the machine. Any processes running .NET 10+ will also include some additional lightweight events describing GC, JIT, and Assembly loading behavior.

```output
Expand Down
16 changes: 9 additions & 7 deletions docs/core/diagnostics/eventpipe.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,18 +25,20 @@ To learn more about the NetTrace format, see the [NetTrace format documentation]

EventPipe is part of the .NET runtime and is designed to work the same way across all the platforms .NET Core supports. This allows tracing tools based on EventPipe, such as `dotnet-counters`, `dotnet-gcdump`, and `dotnet-trace`, to work seamlessly across platforms.

However, because EventPipe is a runtime built-in component, its scope is limited to managed code and the runtime itself. EventPipe events include stacktraces with managed code frame information only. If you want events generated from other unmanaged user-mode libraries, CPU sampling for native code, or kernel events you should use OS-specific tracing tools such as ETW or perf_events. On Linux the [perfcollect tool](./trace-perfcollect-lttng.md) helps automate using perf_events and [LTTng](https://en.wikipedia.org/wiki/LTTng).
However, because EventPipe is a runtime built-in component, its scope is limited to managed code and the runtime itself. Without other tracing tools, EventPipe events include stacktraces with managed code frame information only. To get events from other unmanaged user-mode libraries, CPU sampling for native code, or kernel events, use OS-specific tracing tools such as ETW or perf_events. On Linux, the [perfcollect tool](./trace-perfcollect-lttng.md) helps automate using perf_events and [LTTng](https://en.wikipedia.org/wiki/LTTng).
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use "stack traces" (two words) instead of "stacktraces" for consistency with other diagnostics docs.

Suggested change
However, because EventPipe is a runtime built-in component, its scope is limited to managed code and the runtime itself. Without other tracing tools, EventPipe events include stacktraces with managed code frame information only. To get events from other unmanaged user-mode libraries, CPU sampling for native code, or kernel events, use OS-specific tracing tools such as ETW or perf_events. On Linux, the [perfcollect tool](./trace-perfcollect-lttng.md) helps automate using perf_events and [LTTng](https://en.wikipedia.org/wiki/LTTng).
However, because EventPipe is a runtime built-in component, its scope is limited to managed code and the runtime itself. Without other tracing tools, EventPipe events include stack traces with managed code frame information only. To get events from other unmanaged user-mode libraries, CPU sampling for native code, or kernel events, use OS-specific tracing tools such as ETW or perf_events. On Linux, the [perfcollect tool](./trace-perfcollect-lttng.md) helps automate using perf_events and [LTTng](https://en.wikipedia.org/wiki/LTTng).

Copilot uses AI. Check for mistakes.

Starting in .NET 10, EventPipe on Linux can emit events as [user_events](https://docs.kernel.org/trace/user_events.html), enabling collection of managed events, OS/kernel events, and native callstacks in a single unified trace. This mode requires admin/root privileges and Linux kernel 6.4+. For more information, see [`dotnet-trace collect-linux`](./dotnet-trace.md#dotnet-trace-collect-linux).

Another major difference between EventPipe and ETW/perf_events is admin/root privilege requirement. To trace an application using ETW or perf_events you need to be an admin/root. Using EventPipe, you can trace applications as long as the tracer (for example, `dotnet-trace`) is run as the same user as the user that launched the application.
Comment on lines +30 to 32
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The privilege requirement paragraph now reads like EventPipe never requires admin/root, but the new user_events mode does. Consider clarifying that the "same user" rule applies to the default EventPipe session, while EventPipe (user_events) requires admin/root.

Copilot uses AI. Check for mistakes.

The following table is a summary of the differences between EventPipe and ETW/perf_events.

|Feature|EventPipe|ETW|perf_events|
|-------|---------|---|-----------|
|Cross-platform|Yes|No (only on Windows)|No (only on supported Linux distros)|
|Require admin/root privilege|No|Yes|Yes|
|Can get OS/kernel events|No|Yes|Yes|
|Can resolve native callstacks|No|Yes|Yes|
|Feature|EventPipe|EventPipe (user_events)|ETW|perf_events|
|-------|---------|----------------------|---|-----------|
|Cross-platform|Yes|No (Linux only)|No (only on Windows)|No (only on supported Linux distros)|
Copy link
Member

@noahfalk noahfalk Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
|Cross-platform|Yes|No (Linux only)|No (only on Windows)|No (only on supported Linux distros)|
|Cross-platform|Yes|No (only on supported Linux distros)|No (only on Windows)|No (only on supported Linux distros)|

|Require admin/root privilege|No|Yes|Yes|Yes|
|Can get OS/kernel events|No|Yes|Yes|Yes|
|Can resolve native callstacks|No|Yes|Yes|Yes|

## Use EventPipe to trace your .NET application

Expand Down
Loading