-
Notifications
You must be signed in to change notification settings - Fork 1.2k
updates for ebpf / full host profiling #29270
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,107 @@ | ||||||||||||||||||||||||||||||||||||
--- | ||||||||||||||||||||||||||||||||||||
title: Enabling the eBPF Profiler | ||||||||||||||||||||||||||||||||||||
code_lang: ebpf | ||||||||||||||||||||||||||||||||||||
type: multi-code-lang | ||||||||||||||||||||||||||||||||||||
code_lang_weight: 90 | ||||||||||||||||||||||||||||||||||||
aliases: | ||||||||||||||||||||||||||||||||||||
- /tracing/profiler/enabling/ebpf/ | ||||||||||||||||||||||||||||||||||||
Comment on lines
+3
to
+7
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Let's remove this for now so we can make the page private. |
||||||||||||||||||||||||||||||||||||
further_reading: | ||||||||||||||||||||||||||||||||||||
- link: 'getting_started/profiler' | ||||||||||||||||||||||||||||||||||||
tag: 'Documentation' | ||||||||||||||||||||||||||||||||||||
text: 'Getting Started with Profiler' | ||||||||||||||||||||||||||||||||||||
- link: 'profiler/profiler_troubleshooting/ebpf' | ||||||||||||||||||||||||||||||||||||
tag: 'Documentation' | ||||||||||||||||||||||||||||||||||||
text: 'Fix problems you encounter while using the profiler' | ||||||||||||||||||||||||||||||||||||
--- | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
<div class="alert alert-warning"> | ||||||||||||||||||||||||||||||||||||
<code>ebpf</code> is in beta. Datadog recommends evaluating the profiler in a non-sensitive environment before deploying in production. | ||||||||||||||||||||||||||||||||||||
</div> | ||||||||||||||||||||||||||||||||||||
Comment on lines
+17
to
+19
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
The eBPF profiler is our eBPF-based host profiler built on OpenTelemetry and designed to send profiling data to the Datadog backend via the Datadog Agent. It supports symbolication for compiled languages and is optimized for containerized environments like Docker and Kubernetes. | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
I think a use cases section would be helpful. What do you think? |
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
## Requirements | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
For a summary of the minimum and recommended runtime and tracer versions across all languages, read [Supported Language and Tracer Versions][7]. | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
Supported operating systems | ||||||||||||||||||||||||||||||||||||
: Linux | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
Supported architecture | ||||||||||||||||||||||||||||||||||||
: `amd64` or `arm64` processors | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
Serverless | ||||||||||||||||||||||||||||||||||||
: `ebpf` is not supported on serverless platforms, such as AWS Lambda. | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
OS Settings | ||||||||||||||||||||||||||||||||||||
: TBD | ||||||||||||||||||||||||||||||||||||
Comment on lines
+36
to
+37
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just noting so we don't forget to update or remove this. |
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
Debugging information | ||||||||||||||||||||||||||||||||||||
: Symbols should be available locallly or can be uploaded in CI via `datadog-ci` | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
## Installation | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
The profiler can be used as a standalone executable. | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
### Running the Profiler | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
If the host is running workloads inside containers, it is recommended to run the profiler inside a container as well. A container image is [available][5]. | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
If you're using Kubernetes, follow the instruction for [running in Kubernetes][8]. | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
If you're directly using Docker, follow the instrcutins for [running in Docker][9]. | ||||||||||||||||||||||||||||||||||||
Comment on lines
+49
to
+51
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Just a note, when we GA this, I think we should include these instructions on this page. |
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
If you're not using a container runtime, follow the instructions for running [directly on the host][10] | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
### Configuring the Profiler | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
#### Local symbol upload (Experimental) | ||||||||||||||||||||||||||||||||||||
For compiled languages (C/C++/Rust/Go/...), the profiler can upload local symbols (when available) to Datadog for symbolication. Symbols need to be available locally (unstripped binaries). | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
To enable local symbol upload: | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
1. Set the `DD_HOST_PROFILING_EXPERIMENTAL_UPLOAD_SYMBOLS` environment variable to true. | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
2. Provide a Datadog API key through the `DD_API_KEY` environment variable. | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
3. Provide a Datadog APP key through the `DD_APP_KEY` environment variable. | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
4. Set the `DD_SITE environment` variable to your Datadog site (e.g. datadoghq.com, datadoghq.eu, us5.datadoghq.com, ...). | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Note this will auto-populate based on the site they have selected. |
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
### Build | ||||||||||||||||||||||||||||||||||||
You must first ensure you have the correct version of go installed. In order to build the profiler directly on your machine, you can simply run: | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
```shell | ||||||||||||||||||||||||||||||||||||
make | ||||||||||||||||||||||||||||||||||||
``` | ||||||||||||||||||||||||||||||||||||
Comment on lines
+67
to
+71
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
We should note what the required version of Go is in the Requirements section. Do we know that yet? |
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
### Library | ||||||||||||||||||||||||||||||||||||
TBD | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
### Deploying the shared library | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
TBD | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
#### Installing the library | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
TBD | ||||||||||||||||||||||||||||||||||||
#### Appending a search directory | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
TBD | ||||||||||||||||||||||||||||||||||||
brantigua marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
## Configuration | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
TBD | ||||||||||||||||||||||||||||||||||||
Comment on lines
+73
to
+89
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Let's delete this unless you anticipate us having content to fill it in with before publishing? |
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
## Not sure what to do next? | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
The [Getting Started with Profiler][6] guide takes a sample service with a performance problem and shows you how to use Continuous Profiler to understand and fix the problem. | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
## Further Reading | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
{{< partial name="whats-next/whats-next.html" >}} | ||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
[2]: shttps://github.com/DataDog/dd-otel-host-profiler/releases/ | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||
[3]: https://app.datadoghq.com/profiling | ||||||||||||||||||||||||||||||||||||
[4]: /getting_started/tagging/unified_service_tagging | ||||||||||||||||||||||||||||||||||||
[5]: https://github.com/DataDog/dd-otel-host-profiler/pkgs/container/dd-otel-host-profiler/. | ||||||||||||||||||||||||||||||||||||
[6]: /getting_started/profiler/ | ||||||||||||||||||||||||||||||||||||
[7]: /profiler/enabling/supported_versions/ | ||||||||||||||||||||||||||||||||||||
[8]: https://github.com/DataDog/dd-otel-host-profiler/blob/main/doc/running-in-kubernetes.md | ||||||||||||||||||||||||||||||||||||
[9]: https://github.com/DataDog/dd-otel-host-profiler/blob/main/doc/running-in-docker.md | ||||||||||||||||||||||||||||||||||||
[10]: https://github.com/DataDog/dd-otel-host-profiler/blob/main/doc/running-on-host.md | ||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Adding links for suggested changes in Configuration section. |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -17,8 +17,8 @@ The following tables summarize the features available for each language runtime. | |
|
||
To use the Datadog Profiler, use at least the minimum versions summarized in the following table. For specific profile type availability by version, see [Profile types](#profile-types). | ||
|
||
| | [Java][1] | [Python][2] | [Go][3] | [Ruby][4] | [Node.js][5] | [.NET][6] | [PHP][7] | [Rust/C/C++][8] | | ||
|-----------------------------------|:------------:|:----------------:|:-------------:|:--------------:|:-------------:|:-----------------------------------------------------------------------:|:-------------:|:---------------:| | ||
| | [Java][1] | [Python][2] | [Go][3] | [Ruby][4] | [Node.js][5] | [.NET][6] | [PHP][7] | [Rust/C/C++][8] | [eBPF][9] | | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we want to make this page private, we should note that this is in Preview here or remove it for now. |
||
|-----------------------------------|:------------:|:----------------:|:-------------:|:--------------:|:-------------:|:-----------------------------------------------------------------------:|:-------------:|:---------------:|:---------------:| | ||
| <strong>Minimum runtime version</strong> | [JDK 8+][17] | Python 2.7+ | [previous major Go release][21] | Ruby 2.5+ | Node 14+ | .NET Core 2.1+, .NET 5+, .NET Framework 4.6.1+ | PHP 7.1+ | | | ||
| <strong>Feature-complete runtime version</strong> | [JDK 11+][17] | Python 3.6+ | [latest major Go release][21] | Ruby 3.2+ | Node 18+ | .NET 7+ | PHP 8.0+ | | | ||
| <strong>Feature-complete tracing library version</strong> | [latest][9] | [latest][10] | [latest][11] | [latest][12] | [latest][13] | [latest][14] | [latest][15] | [latest][16] | | ||
|
@@ -28,9 +28,9 @@ To use the Datadog Profiler, use at least the minimum versions summarized in the | |
The following table shows profile type availability by language. For optimal performance and access to all features, Datadog recommends using the latest version of the tracing library for your language. If a specific runtime version isn't indicated, the profile type is available with the minimum runtime version listed in the [Runtime and tracing library versions](#runtime-and-tracing-library-versions). | ||
|
||
|
||
| <div style="width:150px"><div> | [Java][1] | [Python][2] | [Go][3] | [Ruby][4] | [Node.js][5] | [.NET][6] | [PHP][7] | [Rust/C/C++][8] | | ||
|-----------------------------------|:-------------------------------------------------:|:-------:|:------------:|:------:|:---------:|:-------:|:------:|:----------:| | ||
| {{< ci-details title="CPU" >}}The time each function/method spent running on the CPU.{{< /ci-details >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< tooltip glossary="preview" case="title" >}} | | ||
| <div style="width:150px"><div> | [Java][1] | [Python][2] | [Go][3] | [Ruby][4] | [Node.js][5] | [.NET][6] | [PHP][7] | [Rust/C/C++][8] |[eBPF][9] | | ||
|-----------------------------------|:-------------------------------------------------:|:-------:|:------------:|:------:|:---------:|:-------:|:------:|:----------:|:----------:| | ||
| {{< ci-details title="CPU" >}}The time each function/method spent running on the CPU.{{< /ci-details >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< tooltip glossary="preview" case="title" >}} | {{< tooltip glossary="preview" case="title" >}} | | ||
| {{< ci-details title="Exceptions" >}}The number of exceptions raised, including those caught.{{< /ci-details >}} | {{< X >}} | Python 3.7+ | | | | .NET 5+ | {{< X >}} | | | ||
| {{< ci-details title="Allocation" >}}Number and sizes of memory allocations made by each function/method, including allocations which were subsequently freed.{{< /ci-details >}} | [JDK 11+][17] | Python 3.6+ | {{< X >}} | {{< X >}} | | {{< tooltip glossary="preview" case="title" >}}<br>.NET 6+ | {{< X >}} | {{< tooltip glossary="preview" case="title" >}} | | ||
| {{< ci-details title="Heap" >}}The amount of heap memory allocated that remains in use.{{< /ci-details >}} | [JDK 11+][17] | Python 3.6+ | {{< X >}} | alpha<br>Ruby 3.1+ | {{< X >}} | {{< tooltip glossary="preview" case="title" >}}<br>.NET 7+ | | {{< tooltip glossary="preview" case="title" >}} | | ||
|
@@ -42,9 +42,9 @@ The following table shows profile type availability by language. For optimal per | |
|
||
The following table outlines additional profiling features by language. For full functionality and best performance, Datadog recommends using the latest version of your language's tracing library. If a specific runtime version isn't indicated, the feature is available with the minimum runtime version listed in the [Runtime and tracing library versions](#runtime-and-tracing-library-versions). | ||
|
||
| | [Java][1] | [Python][2] | [Go][3] | [Ruby][4] | [Node.js][5] | [.NET][6] | [PHP][7] | [Rust/C/C++][8] | | ||
|-----------------------------------|:-------:|:-------:|:------------:|:------:|:---------:|:-------:|:------:|:----------:| | ||
| {{< ci-details title="Trace to Profiling integration" >}}Find specific lines of code related to performance issues. <a href="/profiler/connect_traces_and_profiles/#identify-code-hotspots-in-slow-traces">Learn more</a>{{< /ci-details >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | | | ||
| | [Java][1] | [Python][2] | [Go][3] | [Ruby][4] | [Node.js][5] | [.NET][6] | [PHP][7] | [Rust/C/C++][8] | [eBPF][9] | | ||
|-----------------------------------|:-------:|:-------:|:------------:|:------:|:---------:|:-------:|:------:|:----------:|:----------:| | ||
| {{< ci-details title="Trace to Profiling integration" >}}Find specific lines of code related to performance issues. <a href="/profiler/connect_traces_and_profiles/#identify-code-hotspots-in-slow-traces">Learn more</a>{{< /ci-details >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | | {{< X >}} | | ||
| {{< ci-details title="Endpoint Profiling" >}}Identify endpoints that are bottlenecks or responsible for heavy resource consumption. <a href="/profiler/connect_traces_and_profiles/#endpoint-profiling">Learn more</a>{{< /ci-details >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | | | ||
| {{< ci-details title="Timeline View" >}}Surface time-based patterns and work distribution over the period of a span. <a href="/profiler/connect_traces_and_profiles/#span-execution-timeline-view">Learn more</a>{{< /ci-details >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | | | ||
| {{< ci-details title="Memory Leaks" >}}A guided workflow to assist in investigating memory leaks. <a href="/profiler/guide/solve-memory-leaks/">Learn more</a>{{< /ci-details >}} | {{< X >}} | | {{< X >}} | | | | | | | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
--- | ||
title: Troubleshooting the eBPF Profiler | ||
code_lang: ebpf | ||
type: multi-code-lang | ||
code_lang_weight: 90 | ||
further_reading: | ||
- link: '/tracing/troubleshooting' | ||
tag: 'Documentation' | ||
text: 'APM Troubleshooting' | ||
--- | ||
|
||
<div class="alert alert-warning"> | ||
<code>eBPF</code> is in Preview. Datadog recommends evaluating the profiler in a non-sensitive environment before deploying in production. | ||
</div> | ||
|
||
|
||
## Further Reading | ||
|
||
{{< partial name="whats-next/whats-next.html" >}} | ||
|
||
[1]: /help/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think about framing this as this as the full host profiler instead of eBPF?