Skip to content

Commit 354e645

Browse files
bd1976brisbd1976bris
committed
[DTLTO][LLVM][Doc] Add DTLTO documentation
1 parent 7274b74 commit 354e645

File tree

2 files changed

+234
-0
lines changed

2 files changed

+234
-0
lines changed

‎llvm/docs/DTLTO.rst

+228
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
===================
2+
DTLTO
3+
===================
4+
.. contents::
5+
:local:
6+
:depth: 2
7+
8+
.. toctree::
9+
:maxdepth: 1
10+
11+
Distributed ThinLTO (DTLTO)
12+
===========================
13+
14+
Distributed ThinLTO (DTLTO) facilitates the distribution of backend ThinLTO
15+
compilations via external distribution systems such as Incredibuild.
16+
17+
The existing method of distributing ThinLTO compilations via separate thin-link,
18+
backend compilation, and link steps often requires significant changes to the
19+
user's build process to adopt, as it requires using a build system which can
20+
handle the dynamic dependencies specified by the index files, such as Bazel.
21+
22+
DTLTO eliminates this need by managing distribution internally within the LLD
23+
linker during the traditional link step. This allows DTLTO to be used with any
24+
build process that supports in-process ThinLTO.
25+
26+
Limitations
27+
-----------
28+
29+
The current implementation of DTLTO has the following limitations:
30+
31+
- The ThinLTO cache is not supported.
32+
- Only ELF and COFF platforms are supported.
33+
- Archives with bitcode members are not supported.
34+
- Only a very limited set of LTO configurations are currently supported, e.g.,
35+
support for basic block sections is not currently available.
36+
37+
Overview of Operation
38+
---------------------
39+
40+
For each ThinLTO backend compilation job, LLD:
41+
42+
1. Generates the required summary index shard.
43+
2. Records a list of input and output files.
44+
3. Constructs a Clang command line to perform the ThinLTO backend compilation.
45+
46+
This information is supplied, via a JSON file, to a distributor program that
47+
executes the backend compilations using a distribution system. Upon completion,
48+
LLD integrates the compiled native object files into the link process.
49+
50+
The design keeps the details of distribution systems out of the LLVM source
51+
code.
52+
53+
Distributors
54+
------------
55+
56+
Distributors are programs responsible for:
57+
58+
1. Consuming the JSON backend compilations job description file.
59+
2. Translating job descriptions into requests for the distribution system.
60+
3. Blocking execution until all backend compilations are complete.
61+
62+
Distributors must return a non-zero exit code on failure. They can be
63+
implemented as binaries or in scripting languages, such as Python. An example
64+
script demonstrating basic local execution is available with the LLVM source
65+
code.
66+
67+
How Distributors Are Invoked
68+
----------------------------
69+
70+
Clang and LLD provide options to specify a distributor program for managing
71+
backend compilations. Distributor options and backend compilation options, can
72+
also be specified. Such options are transparently forwarded.
73+
74+
The backend compilations are currently performed by invoking Clang. For further
75+
details, refer to:
76+
77+
- Clang documentation: https://clang.llvm.org/docs/ThinLTO.html
78+
- LLD documentation: https://lld.llvm.org/DTLTO.html
79+
80+
When invoked with a distributor, LLD generates a JSON file describing the
81+
backend compilation jobs and executes the distributor passing it this file. The
82+
JSON file provides the following information to the distributor:
83+
84+
- The **command line** to execute the backend compilations.
85+
- DTLTO constructs a Clang command line by translating some of the LTO
86+
configuration state into Clang options and forwarding options specified
87+
by the user.
88+
89+
- **Link output path**.
90+
- A string identifying the output to which this LTO invocation will
91+
contribute. Distributors can use this to label build jobs for informational
92+
purposes.
93+
94+
- **Linker's version string**.
95+
- Distributors can use this to determine if the invoked remote optimisation
96+
tool is compatible.
97+
98+
- The list of **imports** required for each job.
99+
- The per-job list of bitcode files from which importing will occur. This is
100+
the same information that is emitted into import files for ThinLTO.
101+
102+
- The **input files** required for each job.
103+
- The per-job set of files required for backend compilation, such as bitcode
104+
files, summary index files, and profile data.
105+
106+
- The **output files** generated by each job.
107+
- The per-job files generated by the backend compilations, such as compiled
108+
object files and toolchain metrics.
109+
110+
Temporary Files
111+
---------------
112+
113+
During its operation, DTLTO generates temporary files. Temporary files are
114+
created in the same directory as the linker's output file and their filenames
115+
include the stem of the bitcode module, or the output file that the LTO
116+
invocation is contributing to, to aid the user in identifying them:
117+
118+
- **JSON Job Description File**:
119+
- Format: `dtlto.<UID>.dist-file.json`
120+
- Example: `dtlto.77380.dist-file.json` (for output file `dtlto.elf`).
121+
122+
- **Object Files From Backend Compilations**:
123+
- Format: `<Module ID stem>.<Task>.<UID>.native.o`
124+
- Example: `my.1.77380.native.o` (for bitcode module `my.o`).
125+
126+
- **Summary Index Shard Files**:
127+
- Format: `<Module ID stem>.<Task>.<UID>.native.o.thinlto.bc`
128+
- Example: `my.1.77380.native.o.thinlto.bc` (for bitcode module `my.o`).
129+
130+
Temporary files are removed, by default, after the backend compilations complete.
131+
132+
JSON Schema
133+
-----------
134+
135+
Below is an example of a JSON job file for backend compilation of the module
136+
`dtlto.o`:
137+
138+
.. code-block:: json
139+
140+
{
141+
"common": {
142+
"linker_output": "dtlto.elf",
143+
"linker_version": "LLD 20.0.0",
144+
"args": [
145+
"/usr/local/clang",
146+
"-O3", "-fprofile-sample-use=my.profdata",
147+
"-o", ["primary_output", 0],
148+
"-c", "-x", "ir", ["primary_input", 0],
149+
["summary_index", "-fthinlto-index=", 0],
150+
"--target=x86_64-sie-ps5"
151+
]
152+
},
153+
"jobs": [
154+
{
155+
"primary_input": ["dtlto.o"],
156+
"summary_index": ["dtlto.1.51232.native.o.thinlto.bc"],
157+
"primary_output": ["dtlto.1.51232.native.o"],
158+
"imports": [],
159+
"additional_inputs": ["my.profdata"]
160+
}
161+
]
162+
}
163+
164+
Key Features of the Schema
165+
~~~~~~~~~~~~~~~~~~~~~~~~~~
166+
167+
- **Input/Output Paths**: Paths are stored in per-file-type array fields. This
168+
allows files to be adjusted, if required, to meet the constraints of the
169+
underlying distribution system. For example, a system may only be able to read
170+
and write remote files to `C:\\sandbox`. The remote paths used can be adjusted
171+
by the distributor for such constraints. Once outputs are back on the local
172+
system, the distributor can rename them as required.
173+
174+
175+
- **Command-Line Template**: Command-line options are stored in a common
176+
template to avoid duplication for each job. The template consists of an array
177+
of strings and arrays. The arrays are placeholders which reference per-job
178+
paths. This allows the remote optimisation tool to be changed without updating
179+
the distributors.
180+
181+
Command-Line Expansion Example
182+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
183+
184+
To create the backend compilation commands, the command-line template is
185+
expanded for each job. Placeholders are expanded in the following way: The first
186+
array element specifies the name of the array field to look in. The remaining
187+
elements are converted to strings and concatenated. Integers are converted by
188+
indexing into the specified array.
189+
190+
The example above generates the following backend compilation command for
191+
`main.o`:
192+
193+
.. code-block:: console
194+
195+
/usr/local/clang -O3 -fprofile-sample-use=my.profdata \
196+
-o dtlto.1.51232.native.o -c -x ir dtlto.o \
197+
-fthinlto-index=dtlto.1.51232.native.o.thinlto.bc --target=x86_64-sie-ps5
198+
199+
This expansion scheme allows the remote optimization tool to be changed without
200+
updating the distributors. For example, if the "args" field in the above example
201+
was replaced with:
202+
203+
.. code-block:: json
204+
205+
"args": [
206+
"custom-codgen-tool",
207+
"-opt-level=2",
208+
"-profile-instrument-use-path=my.profdata",
209+
"-output", ["primary_output", 0],
210+
"-input", ["primary_input", 0],
211+
"-thinlto-index", ["summary_index", 0],
212+
"-triple", "x86_64-sie-ps5"
213+
]
214+
215+
Then distributors can expand the command line without needing to be updated:
216+
217+
.. code-block:: console
218+
219+
custom-codgen-tool -opt-level=2 -profile-instrument-use-path=my.profdata \
220+
-output dtlto.1.51232.native.o -input dtlto.o \
221+
-thinlto-index dtlto.1.51232.native.o.thinlto.bc -triple x86_64-sie-ps5
222+
223+
Constraints
224+
-----------
225+
226+
- Matching versions of Clang and LLD should be used.
227+
- The distributor used must support the JSON schema generated by the version of
228+
LLD in use.

‎llvm/docs/UserGuides.rst

+6
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ intermediate LLVM representation.
3232
DebuggingJITedCode
3333
DirectXUsage
3434
Docker
35+
DTLTO
3536
FatLTO
3637
ExtendingLLVM
3738
GitHub
@@ -164,6 +165,11 @@ Optimizations
164165
This document describes the interface between LLVM intermodular optimizer
165166
and the linker and its design
166167

168+
:doc:`DTLTO`
169+
This document describes the DTLTO implementation, which allows for
170+
distributing ThinLTO backend compilations without requiring support from
171+
the build system.
172+
167173
:doc:`GoldPlugin`
168174
How to build your programs with link-time optimization on Linux.
169175

0 commit comments

Comments
 (0)