You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

320 lines

  1. \chapter{Cross-compi:w
  2. lation toolchains}
  3. \label{cha:cross_compilation_toolchains}
  4. \section{Overall goal}
  5. \label{sec:overall_goal}
  6. Working in the embedded world obviously lead to the use of cross-compilation
  7. toolchains. They are almost always needed, but usually quite specific, and most
  8. of the time, people have two choices: using an existing generic toolchain\footnote{
  9. Such as Linaro's ones:\url{}},
  10. or building their own, which takes a bit of time and knowledge.
  11. \textbf{Buildroot} already solves the knowledge part by providing a ready to use
  12. build system supporting a tremendous amount of combinations allowing one to
  13. build the wanted toolchain without knowing all the details of the process. But
  14. that usually means spending at least half an hour only for building the
  15. toolchain before the real work can begin. This time can quickly increase up to
  16. two hours depending on the computer's performance and the toolchain
  17. configuration.
  18. If we could solve the time problem by pre-building a large variety of
  19. toolchains, combining many attributes, such as the architecture and the C
  20. standard library, with different versions each time, we could cover a large part
  21. of the common cases for which people are used to build their own toolchains.
  22. \subsection{Specifications details}
  23. \label{sub:specifications_details}
  24. Before diving into the details of the toolchain builder itself, it is best to
  25. describe a bit more what the final set of toolchains should look like.
  26. \subsubsection{Architectures}
  27. \label{ssub:architectures}
  28. The targeted architectures have been numerous from the beginning, because most
  29. of them are coming in big-endian and little-endian. Moreover, some of the widely
  30. used architectures, such as ARM or MIPS, come in many flavours, and on top of
  31. that, have a 32 and a 64 bits version.
  32. Those multiple factors led to a quite long list:
  33. \begin{multicols}{3}
  34. \begin{itemize}
  35. \item aarch64
  36. \item aarch64be
  37. \item arcle-750d
  38. \item arcle-hs38
  39. \item armebv7-eabihf
  40. \item armv5-eabi
  41. \item armv6-eabihf
  42. \item armv7-eabihf
  43. \item armv7m
  44. \item bfin
  45. \item m68k-68xxx
  46. \item m68k-coldfire
  47. \item microblazebe
  48. \item microblazeel
  49. \item mips32
  50. \item mips32el
  51. \item mips32r5el
  52. \item mips32r6el
  53. \item mips64
  54. \item mips64el-n32
  55. \item mips64r6el-n32
  56. \item nios2
  57. \item openrisc
  58. \item powerpc-e500mc
  59. \item powerpc64-power8
  60. \item powerpc64le-power8
  61. \item sh-sh4
  62. \item sparc64
  63. \item sparcv8
  64. \item x86-64-core-i7
  65. \item x86-core2
  66. \item x86-i686
  67. \item xtensa-lx60
  68. \end{itemize}
  69. \end{multicols}
  70. \subsubsection{C libraries}
  71. \label{ssub:c_libraries}
  72. There are three common open-source C libraries:
  73. \begin{itemize}
  74. \item glibc: the most common C library, used in most non-embedded platforms\footnote{\url{}}
  75. \item uClibc: a small C library that intends to behave as smaller version of
  76. the glibc\footnote{\url{}}
  77. \item musl: a tiny C library, efficient for example for static linking, or
  78. for quick startup due to less dynamic links\footnote{\url{}}
  79. \end{itemize}
  80. \subsubsection{Two versions for each}
  81. \label{ssub:two_versions_for_each}
  82. As the needs for toolchains are very wide depending on the use-cases, some
  83. people may would like to use a \emph{stable} and reliable version, with less
  84. features, but also less bugs, while others would prefer a more
  85. \emph{bleeding-edge} one, with the latest available features of every possible
  86. software.
  87. These two versions, \emph{stable} and \emph{bleeding-edge}, almost doubled the
  88. number of combinations to produce, which was already too high to manage by hand.
  89. As every combination is not automatically valid\footnote{Indeed, the support for
  90. some architecture may not be complete in or the other piece of software, and
  91. only the build system can tell if a particular configuration will work or not.},
  92. and with the enormous amount of configurations, it was inevitable to have a tool
  93. making the combinations and deciding whether it is a valid.
  94. \section{Developing the builder}
  95. \label{sec:developing_the_builder}
  96. From the very beginning, it was known that the final project would have to make
  97. use of many shell commands, such as many different \verb$make$ calls, and some
  98. \verb$chroot$ or \verb$rsync$. Even if all those tools can be called from other
  99. languages, we also knew that the scripts would not require too much logic.
  100. Finally, as it would probably have to run in some remote containers with more
  101. difficult control over the available libraries, the choice of \textbf{Bash}
  102. would make the overall development less painful, with its ability to run all the
  103. needed commands directly, and the access to all the classical shell tools, such
  104. as \verb$cat$, \verb$grep$, or \verb$set -x$, useful for debugging.
  105. \subsection{Generating the combinations}
  106. \label{sub:generating_the_combinations}
  107. As the goal is to cover a large variety of combinations, this process needs to be
  108. procedural. Just as the \textbf{Linux} kernel, \textbf{Buildroot} uses the
  109. Kconfig system\footnote{\url{}}, which
  110. means that fragments of configuration can be made and combined into the final
  111. \verb$.config$ file.
  112. Unfortunately, not all combinations are valid, and to check which works and
  113. which does not, the use of \textbf{Buildroot} was again required. The final
  114. script thus concatenates the fragments across each and every possibilities, and
  115. runs a \verb$make olddefconfig$\footnote{This \textbf{Buildroot} command
  116. generates a full valid configuration from what is already in \verb$.config$.}
  117. before checking if lines were removed from the originally built fragment,
  118. meaning that it was an invalid fragment.
  119. As expected, with all the wanted fragments in place, this quickly generated a
  120. tremendous amount of possible toolchains, reaching over 130 valid combinations
  121. that then had to be built!
  122. \subsection{First, the builder}
  123. \label{sub:first_the_builder}
  124. One of the main constraints regarding the toolchains, was that they should be
  125. able to run even on quite old operating systems (with old libc bindings), and
  126. thus, they should be built on a quite old operating system too.
  127. The chosen solution was simply to launch the build in a \verb$chroot$\footnote{\url{}}
  128. environment, made with\verb$debootstrap$\footnote{\url{}},
  129. using an old Debian version.
  130. Once the environment in place, building the toolchain is only a matter of
  131. running a few \verb$make$ with the right Kconfig fragments.
  132. Here is the final script:
  133. \url{}
  134. \subsection{Then, the tests}
  135. \label{sub:then_the_tests}
  136. The second constraint was that the toolchains should be automatically tested
  137. before sharing them. As it is not trivial to achieve a hundred percent of
  138. coverage on a toolchain, the focus has been kept on a quite common case:
  139. building a full \textbf{Linux} system, including a kernel, and its root
  140. filesystem, before launching it with \textbf{QEMU}\footnote{\url{}
  141. \\Almost architecture were supported, but there were still a few, like Blackfin
  142. or OpenRISC, that could not be tested that way.}, and checking that the system
  143. reaches userland without any problem. This would at least validate that the
  144. toolchain is stable enough to build a full set of binaries able to boot a
  145. system.
  146. To automatize the launch of the commands, while being able to check their
  147. output, the \verb$expect$ command is greatly helpful. With a script as
  148. simple as "run a, check for b, then run c, etc...", the tool was quickly able to
  149. make the basic boot test, and it is now easy to extend this script to make a lot
  150. more in the booted system.
  151. The final \textbf{expect} script looks like this:
  152. \url{}
  153. \subsection{At last, the packaging}
  154. \label{sub:at_last_the_packaging}
  155. Once the toolchain has been tested, the last stage of the toolchain creation is
  156. the packaging. Not only the binaries must be packed, but also many files around,
  157. like the sources, or the different manifests that \textbf{Buildroot} has
  158. generated. This is also at this moment that the READMEs are generated, before
  159. being included in the archive.
  160. At the end of this packaging process, a massive upload is done with all the
  161. useful files, such as the toolchain archives, but also the different compilation
  162. logs, the tests logs, the manifests as separated files, the checksums, and more
  163. importantly, the sources and the licences of every software used in the
  164. toolchain. This legal aspect is very important since the toolchains are
  165. released to everyone, and do not remain for internal use only.
  166. The upload is made through \textbf{SSH}, using the \verb$rsync$ command, since
  167. it is one more time a simple and reliable tool. Moreover, \verb$rsync$ provides
  168. an interesting feature: the upload is made to a temporary file, and once
  169. complete, a simple \verb$mv$ is done, so that the final file instantly appears
  170. as a whole, and thus, race conditions at the filesystem level are avoided.
  171. The destination folders follow a well defined structure so that it is easy to
  172. browse and manually find a toolchain afterwards. The naming of the toolchains is
  173. also made to allow incremental numbering, so that if an update is made, the
  174. already existing toolchains will not vanish, overwritten by a more recent one.
  175. This allows keeping permanent links, that can be used in scripts, without having
  176. to update them every release.
  177. The next, and obvious step, was about running this whole process in parallel, or
  178. days of CPU time would have to be spent for only one full build.
  179. \subsection{Parallelize the build}
  180. \label{sub:parallelize_the_build}
  181. Free Electrons' CTO, Thomas, as a \textbf{Buildroot} maintainer, had already
  182. searched a bit for CI services that could be able to perform such a build. Since
  183. the \textbf{Buildroot} team already runs continuous integration jobs for their
  184. tool\footnote{This is also about letting \textbf{Buildroot} build a large
  185. quantity of different configurations}, and it is currently running on
  186. \url{}'s CI service\footnote{\url{}
  187. \\ \url{} is the public and free to use instance of the
  188. \textbf{Gitlab} software, hosted and administered by \emph{GitLab, Inc}.}, it
  189. was worth testing it.
  190. The setup only consists of a simple \verb$.gitlab-ci.yaml$ file, that describes
  191. the jobs to do, and it was easy to generate it once a valid toolchain fragment
  192. had been found. As a bonus point, they provide a straightforward way to give the
  193. jobs a private key, meaning the use of \textbf{SSH} was not a problem at all.
  194. This solution quickly proved to work well, and even more, it proved to be very
  195. efficient. Completing a full build (i.e. more than 130 toolchains) in the CI
  196. infrastructure is generally done in a bit more than two hours. This impressive
  197. performance has allowed to do a lot of tests, making the release quite reliable.
  198. For providing such a powerful tool, \url{}'s team surely deserves many
  199. kudos!
  200. \section{A fancy website}
  201. \label{sec:a_fancy_website}
  202. The final step toward a public release was of course developing a website
  203. presenting an easy way to select the toolchains among the many choices.
  204. Moreover, the website should be easy to refresh when a new release is built.
  205. A static website generator using \textbf{Python}, some basic \textbf{Jinja2}
  206. templates and crawling through the filesystem to discover the toolchains, has
  207. been a simple solution, secured, easy to deploy, and flexible enough to do
  208. exactly what was expected.
  209. The basic workflow is to run the generator, giving him the path to the
  210. toolchains' storage, and to the web root folder. The script then walks through
  211. the toolchains and their manifest, to generate a full static website composed
  212. only of \emph{HTML} files, which display really fast, and are by design
  213. protected against many types of web attacks, also making the site reliable and
  214. simple to manage.
  215. The generator is available at this address:
  216. \url{}, and the website is
  217. deployed here: \url{}.
  218. \section{Overall summary}
  219. \label{sec:overall_summary}
  220. The final setup ended with the following chain of operations:
  221. \begin{enumerate}
  222. \item The \verb$$ script is run on the maintainer's
  223. computer, and generates the configurations before pushing to a Gitlab
  224. branch a commit embedding the generated fragments and the corresponding
  225. \verb$.gitlab-ci.yaml$ file.
  226. \item Gitlab thus triggers the different jobs, which are then executed in
  227. parallel on the different available workers.
  228. \item At the end of the jobs, the toolchains are pushed to a storage server.
  229. \item Every minute, the storage server runs a script watching for potential
  230. new toolchains, and if found, refreshes the website.
  231. \end{enumerate}
  232. \vspace*{-10em}
  233. \begin{figure}[H]
  234. \centering
  235. \includegraphics[width=0.35\linewidth]{toolchains-workflow.png}
  236. \caption{Toolchains builder overall workflow.}
  237. \label{fig:toolchains-workflow}
  238. \end{figure}
  239. \section{Release, feedback, and updates}
  240. \label{sec:release_feedback_and_updates}
  241. Through word of mouth in the communities, and a blog post\footnote{\url{}}
  242. on Free Electrons' website, the news of the release quickly spread, and feedback
  243. came very quickly. The overall feeling was quite positive, with many people
  244. sending thankful messages by various means.
  245. Among the messages, some questions were raised, which led to the creation of an
  246. FAQ page on the website, aggregating the most common ones.
  247. With the coming of the next \textbf{Buildroot} release, including, among others
  248. improvements, a more recent GCC, a new release was prepared. But with the
  249. growing number of available toolchains, a new page had to be made, as a
  250. per-architecture summary, presenting all the possible versions, even the old and
  251. deprecated ones. It is thus easy to track any possible software version that is
  252. or was released in the toolchains set.
  253. \section{Final words}
  254. \label{sec:final_words}
  255. Despite not being related to the first subject of the internship, it had many
  256. common points with it, especially in the use of the Continuous Integration.
  257. Furthermore, this allowed me to discover a large amount of uncommon
  258. architectures, and to better understand how toolchains work from the inside.
  259. In the meantime, this also was a nice contribution to the open-source world,
  260. since it is a service that did not exist six months ago, and which is now more
  261. and more used by many different people and projects around the world.