Browse Source

toolchains: finish this part

Signed-off-by: Florent Jacquet <florent.jacquet@free-electrons.com>
master
Skia 2 years ago
parent
commit
84d9844d7c
5 changed files with 458 additions and 13 deletions
  1. BIN
      images/toolchains-workflow.png
  2. +312
    -0
      images/toolchains-workflow.svg
  3. BIN
      report.pdf
  4. +2
    -0
      tex/simple_settings_template.tex
  5. +144
    -13
      tex/toolchains.tex

BIN
images/toolchains-workflow.png View File

Before After
Width: 463  |  Height: 800  |  Size: 66KB

+ 312
- 0
images/toolchains-workflow.svg View File

@@ -0,0 +1,312 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->

<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
width="103.20799mm"
height="178.23813mm"
viewBox="0 0 103.20799 178.23813"
version="1.1"
id="svg8"
inkscape:version="0.92.1 r15371"
sodipodi:docname="toolchains-workflow.svg"
inkscape:export-filename="./toolchains-workflow.png"
inkscape:export-xdpi="114"
inkscape:export-ydpi="114">
<defs
id="defs2">
<marker
inkscape:isstock="true"
style="overflow:visible"
id="marker9937"
refX="0"
refY="0"
orient="auto"
inkscape:stockid="Arrow2Lend">
<path
transform="matrix(-1.1,0,0,-1.1,-1.1,0)"
d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1"
id="path9935"
inkscape:connector-curvature="0" />
</marker>
<marker
inkscape:stockid="Arrow2Lend"
orient="auto"
refY="0"
refX="0"
id="marker6824"
style="overflow:visible"
inkscape:isstock="true"
inkscape:collect="always">
<path
id="path6822"
style="fill:#001200;fill-opacity:1;fill-rule:evenodd;stroke:#001200;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1"
d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
transform="matrix(-1.1,0,0,-1.1,-1.1,0)"
inkscape:connector-curvature="0" />
</marker>
<marker
inkscape:stockid="Arrow2Lend"
orient="auto"
refY="0"
refX="0"
id="marker6790"
style="overflow:visible"
inkscape:isstock="true">
<path
id="path6788"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1"
d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
transform="matrix(-1.1,0,0,-1.1,-1.1,0)"
inkscape:connector-curvature="0" />
</marker>
<marker
inkscape:stockid="Arrow2Lend"
orient="auto"
refY="0"
refX="0"
id="Arrow2Lend"
style="overflow:visible"
inkscape:isstock="true">
<path
id="path4655"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1"
d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
transform="matrix(-1.1,0,0,-1.1,-1.1,0)"
inkscape:connector-curvature="0" />
</marker>
<marker
inkscape:stockid="TriangleOutL"
orient="auto"
refY="0"
refX="0"
id="TriangleOutL"
style="overflow:visible"
inkscape:isstock="true">
<path
id="path4776"
d="M 5.77,0 -2.88,5 V -5 Z"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
transform="scale(0.8)"
inkscape:connector-curvature="0" />
</marker>
</defs>
<sodipodi:namedview
id="base"
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1.0"
inkscape:pageopacity="0.0"
inkscape:pageshadow="2"
inkscape:zoom="0.93967959"
inkscape:cx="78.870737"
inkscape:cy="267.60994"
inkscape:document-units="mm"
inkscape:current-layer="layer1"
showgrid="false"
inkscape:window-width="1918"
inkscape:window-height="1038"
inkscape:window-x="0"
inkscape:window-y="20"
inkscape:window-maximized="1"
fit-margin-top="5"
fit-margin-left="5"
fit-margin-right="5"
fit-margin-bottom="5" />
<metadata
id="metadata5">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<g
inkscape:label="Layer 1"
inkscape:groupmode="layer"
id="layer1"
transform="translate(-54.368374,-7.197811)">
<g
id="g4618"
transform="translate(6.9490077,13.898018)"
style="display:inline">
<rect
y="114.30593"
x="52.770393"
height="51.880959"
width="92.505936"
id="rect4530"
style="fill:#7aff7a;fill-opacity:1;stroke:#007f00;stroke-width:0.70205432;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
<text
id="text4534"
y="123.16654"
x="99.095016"
style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
xml:space="preserve"><tspan
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.05555534px;font-family:Ubuntu;-inkscape-font-specification:Ubuntu;text-align:center;text-anchor:middle;stroke-width:0.26458332px"
y="123.16654"
x="99.095016"
id="tspan4532"
sodipodi:role="line">The storage server</tspan><tspan
id="tspan4536"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.05555534px;font-family:Ubuntu;-inkscape-font-specification:Ubuntu;text-align:center;text-anchor:middle;stroke-width:0.26458332px"
y="136.97537"
x="99.095016"
sodipodi:role="line">watches for new packages</tspan><tspan
id="tspan4538"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.05555534px;font-family:Ubuntu;-inkscape-font-specification:Ubuntu;text-align:center;text-anchor:middle;stroke-width:0.26458332px"
y="150.78419"
x="99.095016"
sodipodi:role="line">and refresh the webpage</tspan></text>
<text
id="text4563"
y="160.42526"
x="99.02417"
style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
xml:space="preserve"><tspan
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:4.93888903px;font-family:Ubuntu;-inkscape-font-specification:Ubuntu;text-align:center;text-anchor:middle;stroke-width:0.26458332px"
y="160.42526"
x="99.02417"
id="tspan4561"
sodipodi:role="line">toolchains.free-electrons.com</tspan></text>
</g>
<g
id="g9925"
style="display:inline">
<text
id="text4622"
y="58.749752"
x="120.90517"
style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
xml:space="preserve"><tspan
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-family:Ubuntu;-inkscape-font-specification:Ubuntu;text-align:center;text-anchor:middle;stroke-width:0.26458332px"
y="58.749752"
x="120.90517"
id="tspan4620"
sodipodi:role="line">PUSH</tspan></text>
<g
id="g9886">
<g
id="g4573"
transform="translate(4.3681437,-0.26727328)">
<rect
y="12.852717"
x="59.617088"
height="36.173721"
width="83.974274"
id="rect4485"
style="fill:#7aff7a;fill-opacity:1;stroke:#007f00;stroke-width:0.77526629;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
<text
id="text4489"
y="20.358492"
x="101.73341"
style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
xml:space="preserve"><tspan
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.05555534px;font-family:Ubuntu;-inkscape-font-specification:Ubuntu;text-align:center;text-anchor:middle;stroke-width:0.26458332px"
y="20.358492"
x="101.73341"
id="tspan4487"
sodipodi:role="line">Generate fragments</tspan><tspan
id="tspan4491"
style="font-size:7.05555534px;text-align:center;text-anchor:middle;stroke-width:0.26458332px"
y="34.167316"
x="101.73341"
sodipodi:role="line"><tspan
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-family:Ubuntu;-inkscape-font-specification:Ubuntu"
id="tspan4522">and</tspan> <tspan
id="tspan4493"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.05555534px;font-family:monospace;-inkscape-font-specification:monospace;text-align:center;text-anchor:middle">.gitlab-ci.yaml</tspan></tspan></text>
<text
id="text4555"
y="45.37669"
x="101.46524"
style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
xml:space="preserve"><tspan
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:4.93888903px;font-family:Ubuntu;-inkscape-font-specification:Ubuntu;text-align:center;text-anchor:middle;stroke-width:0.26458332px"
y="45.37669"
x="101.46524"
id="tspan4553"
sodipodi:role="line">Maintainer's computer</tspan></text>
</g>
<path
style="fill:none;fill-rule:evenodd;stroke:#001200;stroke-width:1;stroke-linecap:square;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker6824);paint-order:stroke markers fill"
d="m 105.94174,48.759165 v 22.72153"
id="path6568"
inkscape:connector-curvature="0"
inkscape:connector-type="polyline" />
</g>
</g>
<g
id="g10009"
style="display:inline">
<g
transform="translate(4.810847,9.5918022)"
id="g4585">
<rect
style="fill:#7aff7a;fill-opacity:1;stroke:#007f00;stroke-width:0.63762075;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
id="rect4514"
width="65.308884"
height="34.038342"
x="68.50708"
y="61.888893" />
<text
xml:space="preserve"
style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
x="101.09206"
y="70.473381"
id="text4518"><tspan
sodipodi:role="line"
id="tspan4516"
x="101.09206"
y="70.473381"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.05555534px;font-family:Ubuntu;-inkscape-font-specification:Ubuntu;text-align:center;text-anchor:middle;stroke-width:0.26458332px">Gitlab CI's workers</tspan><tspan
sodipodi:role="line"
x="101.09206"
y="84.282204"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.05555534px;font-family:Ubuntu;-inkscape-font-specification:Ubuntu;text-align:center;text-anchor:middle;stroke-width:0.26458332px"
id="tspan4520">run the jobs</tspan></text>
<text
xml:space="preserve"
style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
x="89.445862"
y="92.538811"
id="text4559"><tspan
sodipodi:role="line"
id="tspan4557"
x="89.445862"
y="92.538811"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:4.93888903px;font-family:Ubuntu;-inkscape-font-specification:Ubuntu;stroke-width:0.26458332px">gitlab.com</tspan></text>
</g>
<text
id="text4626"
y="115.14215"
x="127.65135"
style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
xml:space="preserve"><tspan
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-family:Ubuntu;-inkscape-font-specification:Ubuntu;text-align:center;text-anchor:middle;stroke-width:0.26458332px"
y="115.14215"
x="127.65135"
id="tspan4624"
sodipodi:role="line">UPLOAD</tspan></text>
<path
inkscape:connection-end="#g4618"
inkscape:connection-start="#g4585"
inkscape:connector-type="polyline"
inkscape:connector-curvature="0"
id="path9927"
d="m 105.97237,105.51904 v 22.68491"
style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker9937)" />
</g>
</g>
</svg>

BIN
report.pdf View File


+ 2
- 0
tex/simple_settings_template.tex View File

@@ -41,6 +41,8 @@
\usepackage{color}
\usepackage{enumitem}

\usepackage{multicol}

%Options: Sonny, Lenny, Glenn, Conny, Rejne, Bjarne, Bjornstrup
\usepackage[Bjornstrup]{fncychap}


+ 144
- 13
tex/toolchains.tex View File

@@ -23,6 +23,94 @@ toolchains, combining many attributes, such as the architecture and the C
standard library, with different versions each time, we could cover a large part
of the common cases where people use to build their own toolchains.

\subsection{Specifications details}
\label{sub:specifications_details}

Before diving into the details of the toolchain builder itself, it is best to
describe a bit more what the final set of toolchains should look like.

\subsubsection{Architectures}
\label{ssub:architectures}

The targeted architectures have been numerous from the beginning, because most
of them are coming in big-endian and little-endian. Moreover, some of the widely
used architecture, such as ARM or MIPS, come with many flavours, and on top of
that, have a 32 and a 64 bits version.

Those multiple factors led to a quite long list:
\begin{multicols}{3}
\begin{itemize}
\item aarch64
\item aarch64be
\item arcle-750d
\item arcle-hs38
\item armebv7-eabihf
\item armv5-eabi
\item armv6-eabihf
\item armv7-eabihf
\item armv7m
\item bfin
\item m68k-68xxx
\item m68k-coldfire
\item microblazebe
\item microblazeel
\item mips32
\item mips32el
\item mips32r5el
\item mips32r6el
\item mips64
\item mips64el-n32
\item mips64r6el-n32
\item nios2
\item openrisc
\item powerpc-e500mc
\item powerpc64-power8
\item powerpc64le-power8
\item sh-sh4
\item sparc64
\item sparcv8
\item x86-64-core-i7
\item x86-core2
\item x86-i686
\item xtensa-lx60
\end{itemize}
\end{multicols}


\subsubsection{C libraries}
\label{ssub:c_libraries}

The C libraries are the three common open-source ones:
\begin{itemize}
\item glibc: the most common C library, used in most non-embedded platforms
\footnote{\url{https://www.gnu.org/software/libc/}}
\item uClibc: a small C library that intends to behave as smaller version of
the glibc \footnote{\url{https://uclibc.org/}}
\item musl: a tiny C library, efficient for example for static linking, or
for quick startup due to less dynamic links
\footnote{\url{https://www.musl-libc.org/}}
\end{itemize}

\subsubsection{Two versions for each}
\label{ssub:two_versions_for_each}

As the needs for toolchains are very wide depending on the use-cases, some
people would like to use a \emph{stable} and reliable version, with less
features, but also less bugs, while others will prefer a more
\emph{bleeding-edge} one, with the latest available features of every possible
software.

These two versions, \emph{stable} and \emph{bleeding-edge}, almost doubled the
number of combinations to produce, which was already too high to manage by hand.

As every combination is not automatically a valid one \footnote{Indeed, the
support for some architecture may not be complete in or the other piece of
software, and only the build system can tell if a particular configuration will
work or not.}, and with the enormous amount of configurations, it was inevitable
to have a tool making the combinations and deciding whether its is a valid one
or not.


\section{Developing the builder}
\label{sec:developing_the_builder}

@@ -165,14 +253,49 @@ Moreover, the website should be easy to refresh when a new release is built.
A static website generator using \textbf{Python}, using some basic
\textbf{Jinja2} templates, and crawling through the filesystem to discover the
toolchains, has been a simple solution, secured, easy to deploy, and flexible
enough to do exactly what was expected. The generator is available at this
address: \url{https://github.com/free-electrons/toolchains-webpage}.
enough to do exactly what was expected.

The basic workflow is to run the generator, giving him the path to the
toolchains' storage, and the path to the web root folder. The script then walks
through the toolchains and their manifest, to generate a full static website
composed only of \emph{HTML} files, which display really fast, and are by design
protected against many types of web attacks, also making the site reliable and
simple to manage.

The generator is available at this address:
\url{https://github.com/free-electrons/toolchains-webpage}, and the website is
deployed here: \url{http://toolchains.free-electrons.com}.

\section{Overall summary}
\label{sec:overall_summary}

The final setup ended with the following chain of operation:
\begin{enumerate}
\item The \verb$update_gitlab-ci.sh$ script is run on the maintainer's
computer, and generates the configurations before pushing to a Gitlab
branch a commit embedding the generated fragments and the corresponding
\verb$.gitlab-ci.yaml$ file.
\item Gitlab thus triggers the different jobs, and they are executed in
parallel on the different available workers.
\item At the end of the jobs, the toolchains are pushed to a storage server.
\item Every minutes, the storage server has a script watching for potential
new toolchains, and if found, refreshes the website.
\end{enumerate}

\vspace*{-10em}
\begin{figure}[H]
\centering
\includegraphics[width=0.35\linewidth]{toolchains-workflow.png}
\caption{Toolchains builder overall workflow.}
\label{fig:toolchains-workflow}
\end{figure}


\section{Release, feedback, and updates}
\label{sec:release_feedback_and_updates}

Through word of mouth in the communities, and a blog post
\footnote{http://free-electrons.com/blog/free-and-ready-to-use-cross-compilation-toolchains/}
\footnote{\url{http://free-electrons.com/blog/free-and-ready-to-use-cross-compilation-toolchains/}}
on Free Electrons' website, the news of the release quickly spread, and feedback
came very quickly. The overall feeling was quite positive, with many people
sending thankful messages though various ways.
@@ -181,14 +304,22 @@ Among the messages, some questions were raised, which led to the creation of an
FAQ page on the website, aggregating the most common ones.

With the coming of the next \textbf{Buildroot} version, including, among others
improvements, a more recent GCC (7), a new release has been prepared. But with the
growing number of available toolchains version, and the differences between the
stable release, and the more bleeding-edge one, the toolchains embedding GCC 6
would disappear from the main page, surrounded by the stable GCC 5, and the
recent GCC 7. Since this would be a common problem in the future, a new page has
been made, as a per-architecture summary, presenting all the possible version,
even the old, deprecated ones.



improvements, a more recent GCC, a new release has been prepared. But with the
growing number of available toolchains, a new page must have been made, as a
per-architecture summary, presenting all the possible version, even the old,
deprecated ones. It is thus easy to track any possible software version that is,
or has been released in the toolchains set.

\section{Final words}
\label{sec:final_words}

Despite not being related to the first subject of the internship, it had many
common points, especially in the use of the Continuous Integration. This allowed
me to discover a large amount of uncommon architectures, and to better
understand how toolchains work from the inside, which I did not expect at all at
first glance.

In the meantime, this has also been a nice contribution to the open-source
world, since it is a service that did not exist six months ago, and which is now
more and more used by many different people and projects around the world.


Loading…
Cancel
Save