|
- \chapter{Custom tests for Continuous Integration}
- \label{cha:custom_tests_for_continuous_integration}
-
- \emph{First of all, it is important to note the distinction between the three
- following elements of the architecture: KernelCI, the custom tests, and the LAVA
- lab. All of them will be detailed in this chapter, but they are all completely
- independent, and are simply working together in Free Electrons' CI
- infrastructure.}
-
- \section{The need for custom tests}
- \label{sec:the_need_for_custom_tests}
-
- \subsection{The KernelCI project}
- \label{sub:the_kernelci_project}
-
- \textbf{KernelCI} is a project started a few years ago that aims at compiling on
- a per hour basis a lot of different upstream \textbf{Linux} kernel trees, then
- sending jobs\footnote{Jobs are the resource \textbf{LAVA} deals with. A more
- detailed explanation can be found in \ref{ssub:the_jobs} at page
- \pageref{ssub:the_jobs}.} using those kernels to the subscribed labs, before
- aggregating the results on different summary pages, where it is easy to see if a
- board has some problems booting \textbf{Linux}, and when the problems started to
- occur.
-
- This project already does a great part of the CI loop by building and displaying
- the results, but still needs the collaboration of the many labs contributing
- their device availability to have the complete process.
-
- With all the different kind of devices provided by the labs from all over the
- world, they have achieved over the years a quite good coverage of the supported
- device types in \textbf{Linux}. But even if they test a great number of
- platforms, the only jobs they send make little more than booting and running a
- few basic commands in userspace.
-
- Jobs as simple as this are not suitable when it comes to test the SATA, the
- USB, or some other specific systems that are usually unused during boot. But
- of course, as \textbf{KernelCI} has to deal with many different labs, they can
- not afford to make the jobs more specific, since many boards, despite being
- identical, may have different configurations and different devices attached to
- it.
-
- To fill that gap, custom tests must be set up at a smaller scale, to comply with
- the specificities of the lab and the devices.
-
- \subsection{Specifications}
- \label{sub:specifications}
-
- As Free Electrons counts many device family maintainers in its engineering team,
- it is particularly important for them to have the finest CI set up for those
- devices. Moreover, they frequently work and develop new drivers on custom
- \textbf{Linux} trees, often coming from the device's vendor tree.
-
- Since this process can take quite a long time, it would be wonderful to have
- some CI process in place for those custom trees, and custom features still being
- in development.
-
- Considering the jobs already run by \textbf{KernelCI}, something more specific
- but still mostly similar was needed to implement the custom tests\footnote{The term
- \emph{custom tests} designates tests sent by a different system than
- \textbf{KernelCI}'s, that would be fully controlled by \textbf{Free Electrons},
- and could thus be tuned finely for the engineers needs.}. Two parts were to be set
- up:
- \begin{itemize}
- \item launching custom scripts on the devices to check the specific
- functionalities.
- \item building and booting custom kernel trees that are not already taken
- care of by \textbf{KernelCI}.
- \end{itemize}
-
- Of course, once both parts are running, they can be combined to launch custom
- scripts on custom kernels.
-
- Last but not least, the overall architecture should be supporting two
- operating modes:
- \begin{itemize}
- \item a \textbf{manual} one, that could be triggered by hand, to run only
- the asked specific tests with a user-built kernel, and give immediate
- report once the job has been run.
- \item an \textbf{automatic} one, that would run tests every day, and make
- daily reports to the maintainers.
- \end{itemize}
-
-
- \section{The CI lab}
- \label{sec:the_ci_lab}
-
- Applying continuous integration to an operating system is not an easy task.
- Since it needs external hardware management, power supply control, and
- input/output control, it clearly requires a complex infrastructure.
-
- Free Electrons has a lab now for more than one year, that it is running fine,
- continuously testing more than 35 devices, and reporting public results
- making them available for the \textbf{Linux} community.
-
- \subsection{The hardware part}
- \label{sub:the_hardware_part}
-
- Since the tested software is an operating system, it needs to run on real
- hardware, and thus, it differs from more usual CI that typically runs in some
- container technology.
-
- Built last year, the farm at Free Electrons takes the form of a big cabinet,
- with eight drawers, capable of storing up to 50 boards. With those devices, USB
- hubs, switches, and ATX power supply with their TCP controlled relays can be
- found in each and every drawer. For the main management, the farm also hosts a
- main USB hub, a main switch, and most importantly, the NUC that runs parts of
- the software driving the whole lab.
-
- Everything is well wired and labelled to keep the material in a maintainable
- state.
-
- For more information about the ins and outs of this part, one can read this blog
- article:
- \url{http://free-electrons.com/blog/hardware-infrastructure-free-electrons-lab/}.
-
- \begin{figure}[H]
- \centering
- \includegraphics[height=0.4\paperheight]{lab.JPG}
- \caption{An overview of the cabinet hosting the hardware part of the lab.}
- \label{fig:lab}
- \end{figure}
-
- \subsection{The software: LAVA}
- \label{sub:the_lava_software}
-
- In order to run everything from a software point of view, \textbf{LAVA} has been
- used from the very beginning. LAVA stands for \emph{Linaro Automated Validation
- Architecture}, and provides a quite good way to manage and schedule automated
- tests on embedded devices. Most of the job to do to implement custom tests was
- about using \textbf{LAVA} the right way to perform the tests, so this part will
- include more details than the hardware one.
-
- \subsubsection{LAVA: v1 or v2?}
- \label{ssub:lava_v1_or_v2}
-
- Since \textbf{LAVA} is currently in the middle of a huge refactoring of its
- internal workflow and exposed API, we usually find people talking about
- \textbf{LAVA v1} and \textbf{LAVA v2}. It is actually the same software, with
- different behaviors at many levels, but with the same final goal: running what
- is called a job, a user described list of actions, on a device, and reporting
- the events that happened.
-
- At the beginning of the internship, only the first version was in use, and part
- of the job was about migrating to the second one. This ran through a lot of
- different problems and bugs that needed to be fixed before taking the next step,
- but we finally ended up with a fresh and running architecture using mostly
- \textbf{LAVA v2}.
-
- For clarity's sake, only the \textbf{final setup} will be described in this report. Those
- interested in the original software architecture can still read this blog post:
- \url{http://free-electrons.com/blog/software-architecture-free-electrons-lab/}.
-
- \subsubsection{The jobs}
- \label{ssub:the_jobs}
-
- The main resource \textbf{LAVA} has to deal with is the job. A job is defined by
- a \textbf{YAML}\footnote{\url{http://yaml.org/}} structure, describing multiple
- sections:
- \begin{itemize}
- \item The \textbf{device-type}, which is the name of the device you want to
- run the test on. \\
- \emph{Examples: beaglebone-black, sun8i-h3-orangepi-pc, ...}
- \item The \textbf{boot method}, to tell \textbf{LAVA} which method should be
- used to boot the device. \\
- \emph{Examples: \textbf{fastboot} or \textbf{U-boot}, \textbf{NFS} or
- ramdisk, \textbf{SSH}, ...}
- \item The \textbf{artifacts} URLs. This includes the \emph{kernel}, the
- \emph{device tree}, the \emph{modules}, and the \emph{rootfs}. Only the
- kernel is completely mandatory to boot the boards, but the other ones
- are common for almost every non-exotic device.
- \item The \textbf{tests} to run once the device is booted. This can include
- many possibilities, since it generally points to some shell scripts to
- be executed as root in userspace. It is the main place to customize the
- tests.
- \item Other less important sections, such as some \textbf{metadata}, the
- \textbf{notifications}, or custom \textbf{timeouts}.
- \end{itemize}
-
- Once a job has been sent either by the web interface or by the API, it is queued
- until a device corresponding to the asked device-type is free to be used. Then
- the job gets scheduled and run, before finishing either with the
- \textbf{Complete} status when everything ran well, or the \textbf{Incomplete}
- status when there was a problem during the execution of the different tasks.
-
- When a test is complete, \textbf{LAVA} provides access to the results by many
- ways, such as the web UI, the API, some emails, or a callback system making the
- job able to push its results to some other APIs.
-
- \subsubsection{A distributed system}
- \label{ssub:a_distributed_system}
-
- \textbf{LAVA} has been designed to scale for far more complex labs than Free
- Electrons' one. It is thus split into two parts: a master, and a worker.
-
- \begin{description}
- \item[The master]: \\
- The master can exist in only instance. It is in charge of three tasks:
- \begin{itemize}
- \item The \textbf{web interface}, allowing in-browser interaction
- with the software.
- \item The job \textbf{scheduler}, responsible for sending the queued
- jobs to the available devices.
- \item The \textbf{dispatcher-master}, that manages the different
- possible workers, and sends jobs to them.
- \end{itemize}
- The master also has the connection to the relational database.
- \item[The worker]: \\
- The worker is divided in only two parts:
- \begin{itemize}
- \item The \textbf{slave} is the part that connects to the
- dispatcher-master and receives the jobs to run.
- \item The \textbf{dispatcher} is the only part that really interacts
- with the devices under test. It is spawned on demand by the
- slave when a job needs to be run. It is also the only part that
- does not run as a daemon.
- \end{itemize}
- \end{description}
-
- \begin{figure}[H]
- \centering
- \includegraphics[width=0.8\linewidth]{arch-overview.png}
- \caption{Architecture schematic of the \textbf{LAVA} software.}
- \label{fig:arch-overview}
- \end{figure}
-
-
- \section{Developing custom tests}
- \label{sec:developing_custom_tests}
-
- \subsection{Beginning a proof of concept}
- \label{sub:beginning_a_proof_of_concept}
-
- At the beginning, a basic and functional, but still blurred specification was
- made, but it required a proof-of-concept to see how it would fit in final
- production. It had quickly been named \textbf{CTT}, standing for \emph{Custom
- Test Tool}\footnote{You can find the sources at this address:
- \url{https://github.com/free-electrons/custom_tests_tool}}, and that is how the
- software building the custom jobs will be designated till the end of this
- report.
-
- The choice of \textbf{Python} was more than obvious, since this language is accessible
- and widely used in the embedded world for its flexibility and portability.
- Moreover, most of the Free Electrons engineers had already used it, and it was
- not an option to introduce a new, unknown technology, in an architecture they
- would have to maintain in the future.
-
- \subsubsection{Understanding the depth of LAVA's jobs}
- \label{ssub:understanding_the_lava_jobs}
-
- The first simple part was about \textbf{LAVA}. Since \textbf{KernelCI} already
- provides everything (kernel, dtb and rootfs) needed to run a successful job in
- \textbf{LAVA}, the only part remaining was crafting and sending jobs.
-
- An easy and simple, yet flexible solution, was to use a template engine to
- parametrize a generic job written once and for all.
-
- The job syntax is using the human friendly \textbf{YAML}, but even if it is
- readable and easy to write, the data structure itself required by \textbf{LAVA}
- is a bit complex, and it is thus truly inconvenient to write tests by hand.
-
- Once filled, the template would just have to be sent to \textbf{LAVA} through
- its XML-RPC\footnote{\url{https://en.wikipedia.org/wiki/XML-RPC}} API to create
- a job.
-
- Knowing what to put in that template was one of the most interesting moment of
- this part, since it was like discovering a new programming language. There are
- always new features to discover, and new mechanisms for using them, and finally
- to make \textbf{LAVA} do exactly what you want. It is also during this period
- that most of the migration to \textbf{LAVA v2} was prepared, meaning that the
- configuration of the different levels of \textbf{LAVA} was altered.
-
- It was often required to discuss with the \textbf{LAVA} community, on
- \url{irc://irc.freenode.net#linaro-lava}, to get clarification when the
- documentation happened to be incomplete, or when \textbf{LAVA} needed to be
- improved\footnote{See these patches for example: \\
- \url{https://git.linaro.org/lava/lava-dispatcher.git/commit/?id=8df17dd7355cd82f37e1ef22a6c9d88ede44f650} \\
- \url{https://git.linaro.org/lava/lava-dispatcher.git/commit/?id=3bfdcdb672f1a15da96bbb221a26847dd6bf2865} \\
- Also don't hesitate to run \verb$git log --author "Florent Jacquet"$ in the
- \verb$lava-dispatcher$ and the \verb$lava-server$ projects to get an overview of
- the contributions made to \textbf{LAVA} (Also available in appendix
- \ref{cha:list_of_contributions_to_lava}, page
- \pageref{cha:list_of_contributions_to_lava}).
- }.
-
- \subsubsection{Using custom artifacts}
- \label{ssub:using_custom_artifacts}
-
- Once we could easily send jobs, the next step was about sending custom
- artifacts, such as the user-built kernel. This would be useful for the first
- manual mode of the tool, when a user would launch some jobs from his
- workstation, using a kernel built from one of his working trees.
-
- \textbf{LAVA} allowing the use of files local to the dispatcher, it would be a
- really convenient solution to provide the artifacts without setting up some
- \textbf{FTP} server or other complicated means of serving files.
-
- \textbf{SSH}, with the \verb$scp$ command, allows efficient and reliable file transfers
- between two machines, and since the engineers have an easy access to the
- dispatcher using one of Free Electrons' VPNs\footnote{Virtual Private Network
- (\url{https://en.wikipedia.org/wiki/Virtual\_private\_network})}, it would be
- easy to give them permissions to send files.
-
- With \textbf{Python}, the \textbf{paramiko}\footnote{\url{http://www.paramiko.org/}}
- library, allowing a native use of \textbf{SSH}, makes the choice of that
- protocol even more comfortable.
-
- \subsubsection{Launching automatic jobs}
- \label{ssub:crawling_for_existing_artifacts}
-
- The other mode of the tool, as an automatic launcher, would require to fetch
- pre-built artifacts available from a remote location, such as
- \textbf{KernelCI}'s storage, or Free Electrons' once the custom builds would be
- set up. Fortunately, the \textbf{KernelCI} website also provides an API,
- allowing to retrieve their latest builds.
-
- The most difficult part was then to make sure that the crawler would have enough
- information about the boards to fetch their specific artifacts, while trying to
- avoid having a very big file storing every possible data about the boards.
- Once done, getting the artifacts would only be a matter of crafting the right
- URL.
-
- This ended up with a simple \textbf{JSON}\footnote{\url{http://json.org/}}
- file, storing the list of the boards, each storing four strings:
- \begin{itemize}
- \item \textbf{arch}, the architecture of the board, to guess which kernel to
- use.
- \item \textbf{dt}, the device-tree, also mandatory for booting the devices,
- and unique to each and every one of them.
- \item \textbf{rootfs}, since they are built for many architecture flavours
- (ARMv4, ARMv5, ARMv7, and ARMv8)
- \item \textbf{test\_plan}, since it is mandatory for \textbf{LAVA}, and must
- be configured on a per device basis.
- \end{itemize}
-
- This file proved to be simple enough, and the crafter's job is now only about
- crafting an URL, and checking if the artifact actually exists.
-
- With the crawlers done, and the rest of the tool already working, the only thing
- that remained to be done in \textbf{CTT} was the custom scripts to be run once
- the userspace is reached.
-
- \subsection{Running custom scripts}
- \label{sub:running_custom_scripts}
-
- \subsubsection{Writing a test suite}
- \label{ssub:writing_a_test_suite}
-
- Among the many possibilities brought by the \textbf{LAVA} job structure, is the
- possibility of designating a \emph{git} repository and a path in that repo to a
- file that would be executed automatically by \textbf{LAVA} from the device's
- userland shell.
-
- Before writing more complex tests which would require some time of development,
- a simple \verb$echo "Hello world!"$ made just the job. This allowed to do a lot
- of testing, checking all possible solutions, and finally define an architecture
- that would be both simple and functional enough for the custom tests' needs.
-
- \subsubsection{Integrating custom tools in the root file system}
- \label{ssub:integrating_custom_tools_in_the_rootfs}
-
- Before writing more advanced test scripts in the test suite, a problem had to be
- solved. Many of the tests would require tools or commands that are not shipped
- by default in \textbf{KernelCI}'s rootfs. Moreover, a requirement was that this
- rootfs should be compiled for each ARM flavour, unlike the extremely generic one
- built by \textbf{KernelCI}.
-
- An easy and flexible way of building custom root filesystems is to use
- \textbf{Buildroot}\footnote{\url{https://buildroot.org/}}. This led to some
- simple glue scripts\footnote{\url{https://github.com/free-electrons/buildroot-ci}}
- building the few configurations requested by the farm, which are mainly
- including \emph{iperf}\footnote{\url{https://en.wikipedia.org/wiki/Iperf}
- and \url{https://iperf.fr/}} and a full \emph{ping}\footnote{One that includes
- the \verb$-f$ option, for ping floods.} version for network stressing, and
- \emph{Bonnie++}\footnote{\url{https://en.wikipedia.org/wiki/Bonnie++} and
- \url{http://www.coker.com.au/bonnie++/}} for filesystem performances, over a
- classic Busybox\footnote{\url{https://en.wikipedia.org/wiki/BusyBox} and
- \url{https://busybox.net/}} that provides the rest of the system.
-
- As my first experience with \textbf{Buildroot}, this was a quick but interesting
- part that made me discover the power of build systems in the embedded world.
-
- \subsubsection{The road to complex jobs}
- \label{ssub:the_road_to_complex_jobs}
-
- With a test suite and custom root filesystem, the overall architecture was in
- place. To verify that everything would work as expected, more complex tests were
- to be written.
-
- As Busybox provides only \emph{Ash} as its default shell, the scripts needed to
- be compatible with this software, and thus could not take advantage of some Bash
- features. This turned out to be quite an exercise, since most of the OSs in 2017
- provide the latter by default, and the differences may in some cases cause some
- headache finding workarounds for complex operations.
-
- The other most interesting part was the development of the first
- \emph{Multinode job}\footnote{\url{https://validation.linaro.org/static/docs/v2/multinodeapi.html}}.
- This is the \textbf{LAVA} term to describe jobs that require multiple devices,
- such as some network-related jobs. Since the boards need to interact, they need
- to be synchronized, and \textbf{LAVA} provides some tools in the runtime
- environment to allow data exchanges between the devices, but as with classic
- threads or processes, this can quickly lead to some race conditions, deadlocks,
- or other interesting concurrency problematics.
-
- Once all those problems addressed, with the network tests running, a little
- presentation to the team was given, so that everyone would know the status
- of the custom Continuous Integration, and this also allowed to show them the
- architecture so that they could easily add new boards and tests in the future.
-
- \subsection{Adding some reporting}
- \label{sub:adding_some_reporting}
-
- With the two operating modes of \textbf{CTT}, came two modes of reporting: one
- for the manual tests, and the other for the daily ones.
-
- For the first and easy part, it was just about adding the correct \emph{notify}
- section to the jobs template, so that when an engineer sends a job manually to
- \textbf{LAVA}, his email address is included in the definition and he gets a
- message as soon as the job is finished, with some details about what worked and
- what failed.
-
- For the second part, the daily tests, the need was to aggregate the results of
- the past twenty-four hours into a single, personalized email. Indeed, each
- engineer can subscribe to some devices, and in order not to make the reporting
- too verbose, a script builds a specific email for every one, so that people only
- get the results they are interested in.
-
- \subsection{Integrating custom kernels}
- \label{sub:integrating_custom_kernels}
-
- \subsubsection{Goal}
- \label{ssub:goal}
-
- The next and last step toward fully customized CI tests, was building custom
- kernels. Just like \textbf{KernelCI} does every hour, the goal is to monitor a
- list of kernel trees, pull them, then build them with specific configurations,
- and store the artifacts online, so that \textbf{LAVA} could easily use them.
-
- Custom kernels really come in handy in two cases. When the engineers would like
- to follow a specific tree they work on, but this tree is not close enough to
- mainline and \textbf{KernelCI} does not track it, Free Electrons' builder would
- be in charge of it. The other useful case, is when a test requires custom kernel
- configuration, such as the activation of the hardware cryptographic modules,
- that are platform specific, thus not in the range of \textbf{KernelCI}'s builds.
-
- \subsubsection{Setting up a kernel builder}
- \label{ssub:setting_up_a_kernel_builder}
-
- Mainly based on \textbf{KernelCI}'s Jenkins scripts, but with some
- modifications to work in standalone, the builder\footnote{\url{https://github.com/free-electrons/kernel-builder}}
- is split in two parts: a first script that checks the trees and prepares
- tarballs of the sources when needed, and a second script that builds the
- prepared sources.
-
- In the overall CI architecture, the two scripts are called sequentially, just
- before launching \textbf{CTT} in automatic mode, so that the newly created
- kernel images can quickly be tested. Of course this required adding to
- \textbf{CTT} the requested logic to crawl either on \textbf{KernelCI}'s storage,
- or Free Electrons' one.
-
- \subsection{A full rework before the end}
- \label{sub:a_full_rework_before_the_end}
-
- Before the end of the internship, everything was fully operational, up and
- running, but a quite huge problem remained. Indeed, the whole code of
- \textbf{CTT} had been developed quickly, as a proof-of-concept, and even if the
- technological choices were not bad, the overall design of the software made it
- awful to maintain.
-
- The decision was taken, as about one month remained, to rework completely the
- tool, so that it would be easier in the future to add new features. The
- technical debt brought by the proof-of-concept design pattern would also be
- paid.
-
- One of the engineers had already taken time to rework small parts, but was
- keeping the internal API untouched when some functions or classes required to be
- split in multiple ones. More was needed, but still, he had quite a good vision
- of the tool's design, and greatly helped in its refactoring.
-
- This brought along the way many interesting side-effects: unit tests to almost
- all the newly created classes, flexible and modular design, simpler
- configuration files, better user-interface, improved safety regarding the
- crafted jobs, and a fully rewritten README.
-
- Despite not being originally planned in the main subject of the internship, this
- truly was an instructive part, since it was all about software design, and
- making choices that would help make the tool maintainable for the long term, and
- not something that would fall into oblivion in less than six months.
|