ST50/tex/ci_tests.tex

\chapter{Custom tests for Continuous Integration}
\label{cha:custom_tests_for_continuous_integration}

\emph{First of all, it is important to note the distinction between the three
following elements of the architecture: KernelCI, the custom tests, and the LAVA
lab. All of them will be detailed in this chapter, but they are all completely
independent, and are simply working together in Free Electrons' CI
infrastructure.}

\section{The need for custom tests}
\label{sec:the_need_for_custom_tests}

\subsection{The KernelCI project}
\label{sub:the_kernelci_project}

\textbf{KernelCI} is a project started a few years ago that aims at compiling on
a per hour basis a lot of different upstream \textbf{Linux} kernel trees, then
sending jobs\footnote{Jobs are the resource \textbf{LAVA} deals with. A more
detailed explanation can be found in \ref{ssub:the_jobs} at page
\pageref{ssub:the_jobs}.} using those kernels to the subscribed labs, before
aggregating the results on different summary pages, where it is easy to see if a
board has some problems booting \textbf{Linux}, and when the problems started to
occur.

This project already does a great part of the CI loop by building and displaying
the results, but still needs the collaboration of the many labs contributing
their device availability to have the complete process.

With all the different kind of devices provided by the labs from all over the
world, they have achieved over the years a quite good coverage of the supported
device types in \textbf{Linux}. But even if they test a great number of
platforms, the only jobs they send make little more than booting and running a
few basic commands in userspace.

Jobs as simple as this are not suitable when it comes to test the SATA, the
USB, or some other specific systems that are usually unused during boot. But
of course, as \textbf{KernelCI} has to deal with many different labs, they can
not afford to make the jobs more specific, since many boards, despite being
identical, may have different configurations and different devices attached to
it.

To fill that gap, custom tests must be set up at a smaller scale, to comply with
the specificities of the lab and the devices.

\subsection{Specifications}
\label{sub:specifications}

As Free Electrons counts many device family maintainers in its engineering team,
it is particularly important for them to have the finest CI set up for those
devices.  Moreover, they frequently work and develop new drivers on custom
\textbf{Linux} trees, often coming from the device's vendor tree.

Since this process can take quite a long time, it would be wonderful to have
some CI process in place for those custom trees, and custom features still being
in development.

Considering the jobs already run by \textbf{KernelCI}, something more specific
but still mostly similar was needed to implement the custom tests\footnote{The term
\emph{custom tests} designates tests sent by a different system than
\textbf{KernelCI}'s, that would be fully controlled by \textbf{Free Electrons},
and could thus be tuned finely for the engineers needs.}. Two parts were to be set
up:
\begin{itemize}
    \item launching custom scripts on the devices to check the specific
    functionalities.
    \item building and booting custom kernel trees that are not already taken
    care of by \textbf{KernelCI}.
\end{itemize}

Of course, once both parts are running, they can be combined to launch custom
scripts on custom kernels.

Last but not least, the overall architecture should be supporting two
operating modes:
\begin{itemize}
    \item a \textbf{manual} one, that could be triggered by hand, to run only
        the asked specific tests with a user-built kernel, and give immediate
        report once the job has been run.
    \item an \textbf{automatic} one, that would run tests every day, and make
        daily reports to the maintainers.
\end{itemize}


\section{The CI lab}
\label{sec:the_ci_lab}

Applying continuous integration to an operating system is not an easy task.
Since it needs external hardware management, power supply control, and
input/output control, it clearly requires a complex infrastructure.

Free Electrons has a lab now for more than one year, that it is running fine,
continuously testing more than 35 devices, and reporting public results
making them available for the \textbf{Linux} community.

\subsection{The hardware part}
\label{sub:the_hardware_part}

Since the tested software is an operating system, it needs to run on real
hardware, and thus, it differs from more usual CI that typically runs in some
container technology.

Built last year, the farm at Free Electrons takes the form of a big cabinet,
with eight drawers, capable of storing up to 50 boards. With those devices, USB
hubs, switches, and ATX power supply with their TCP controlled relays can be
found in each and every drawer. For the main management, the farm also hosts a
main USB hub, a main switch, and most importantly, the NUC that runs parts of
the software driving the whole lab.

Everything is well wired and labelled to keep the material in a maintainable
state.

For more information about the ins and outs of this part, one can read this blog
article:
\url{http://free-electrons.com/blog/hardware-infrastructure-free-electrons-lab/}.

\begin{figure}[H]
    \centering
    \includegraphics[height=0.4\paperheight]{lab.JPG}
    \caption{An overview of the cabinet hosting the hardware part of the lab.}
    \label{fig:lab}
\end{figure}

\subsection{The software: LAVA}
\label{sub:the_lava_software}

In order to run everything from a software point of view, \textbf{LAVA} has been
used from the very beginning. LAVA stands for \emph{Linaro Automated Validation
Architecture}, and provides a quite good way to manage and schedule automated
tests on embedded devices. Most of the job to do to implement custom tests was
about using \textbf{LAVA} the right way to perform the tests, so this part will
include more details than the hardware one.

\subsubsection{LAVA: v1 or v2?}
\label{ssub:lava_v1_or_v2}

Since \textbf{LAVA} is currently in the middle of a huge refactoring of its
internal workflow and exposed API, we usually find people talking about
\textbf{LAVA v1} and \textbf{LAVA v2}. It is actually the same software, with
different behaviors at many levels, but with the same final goal: running what
is called a job, a user described list of actions, on a device, and reporting
the events that happened.

At the beginning of the internship, only the first version was in use, and part
of the job was about migrating to the second one. This ran through a lot of
different problems and bugs that needed to be fixed before taking the next step,
but we finally ended up with a fresh and running architecture using mostly
\textbf{LAVA v2}.

For clarity's sake, only the \textbf{final setup} will be described in this report. Those
interested in the original software architecture can still read this blog post:
\url{http://free-electrons.com/blog/software-architecture-free-electrons-lab/}.

\subsubsection{The jobs}
\label{ssub:the_jobs}

The main resource \textbf{LAVA} has to deal with is the job. A job is defined by
a \textbf{YAML}\footnote{\url{http://yaml.org/}} structure, describing multiple
sections:
\begin{itemize}
    \item The \textbf{device-type}, which is the name of the device you want to
        run the test on. \\
        \emph{Examples: beaglebone-black, sun8i-h3-orangepi-pc, ...}
    \item The \textbf{boot method}, to tell \textbf{LAVA} which method should be
        used to boot the device. \\
        \emph{Examples: \textbf{fastboot} or \textbf{U-boot}, \textbf{NFS} or
        ramdisk, \textbf{SSH}, ...}
    \item The \textbf{artifacts} URLs. This includes the \emph{kernel}, the
        \emph{device tree}, the \emph{modules}, and the \emph{rootfs}. Only the
        kernel is completely mandatory to boot the boards, but the other ones
        are common for almost every non-exotic device.
    \item The \textbf{tests} to run once the device is booted. This can include
        many possibilities, since it generally points to some shell scripts to
        be executed as root in userspace. It is the main place to customize the
        tests.
    \item Other less important sections, such as some \textbf{metadata}, the
        \textbf{notifications}, or custom \textbf{timeouts}.
\end{itemize}

Once a job has been sent either by the web interface or by the API, it is queued
until a device corresponding to the asked device-type is free to be used. Then
the job gets scheduled and run, before finishing either with the
\textbf{Complete} status when everything ran well, or the \textbf{Incomplete}
status when there was a problem during the execution of the different tasks.

When a test is complete, \textbf{LAVA} provides access to the results by many
ways, such as the web UI, the API, some emails, or a callback system making the
job able to push its results to some other APIs.

\subsubsection{A distributed system}
\label{ssub:a_distributed_system}

\textbf{LAVA} has been designed to scale for far more complex labs than Free
Electrons' one. It is thus split into two parts: a master, and a worker.

\begin{description}
    \item[The master]: \\
        The master can exist in only instance. It is in charge of three tasks:
        \begin{itemize}
            \item The \textbf{web interface}, allowing in-browser interaction
                with the software.
            \item The job \textbf{scheduler}, responsible for sending the queued
                jobs to the available devices.
            \item The \textbf{dispatcher-master}, that manages the different
                possible workers, and sends jobs to them.
        \end{itemize}
        The master also has the connection to the relational database.
    \item[The worker]: \\
        The worker is divided in only two parts:
        \begin{itemize}
            \item The \textbf{slave} is the part that connects to the
                dispatcher-master and receives the jobs to run.
            \item The \textbf{dispatcher} is the only part that really interacts
                with the devices under test. It is spawned on demand by the
                slave when a job needs to be run. It is also the only part that
                does not run as a daemon.
        \end{itemize}
\end{description}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.8\linewidth]{arch-overview.png}
    \caption{Architecture schematic of the \textbf{LAVA} software.}
    \label{fig:arch-overview}
\end{figure}


\section{Developing custom tests}
\label{sec:developing_custom_tests}

\subsection{Beginning a proof of concept}
\label{sub:beginning_a_proof_of_concept}

At the beginning, a basic and functional, but still blurred specification was
made, but it required a proof-of-concept to see how it would fit in final
production. It had quickly been named \textbf{CTT}, standing for \emph{Custom
Test Tool}\footnote{You can find the sources at this address:
\url{https://github.com/free-electrons/custom_tests_tool}}, and that is how the
software building the custom jobs will be designated till the end of this
report.

The choice of \textbf{Python} was more than obvious, since this language is accessible
and widely used in the embedded world for its flexibility and portability.
Moreover, most of the Free Electrons engineers had already used it, and it was
not an option to introduce a new, unknown technology, in an architecture they
would have to maintain in the future.

\subsubsection{Understanding the depth of LAVA's jobs}
\label{ssub:understanding_the_lava_jobs}

The first simple part was about \textbf{LAVA}. Since \textbf{KernelCI} already
provides everything (kernel, dtb and rootfs) needed to run a successful job in
\textbf{LAVA}, the only part remaining was crafting and sending jobs.

An easy and simple, yet flexible solution, was to use a template engine to
parametrize a generic job written once and for all.

The job syntax is using the human friendly \textbf{YAML}, but even if it is
readable and easy to write, the data structure itself required by \textbf{LAVA}
is a bit complex, and it is thus truly inconvenient to write tests by hand.

Once filled, the template would just have to be sent to \textbf{LAVA} through
its XML-RPC\footnote{\url{https://en.wikipedia.org/wiki/XML-RPC}} API to create
a job.

Knowing what to put in that template was one of the most interesting moment of
this part, since it was like discovering a new programming language. There are
always new features to discover, and new mechanisms for using them, and finally
to make \textbf{LAVA} do exactly what you want. It is also during this period
that most of the migration to \textbf{LAVA v2} was prepared, meaning that the
configuration of the different levels of \textbf{LAVA} was altered.

It was often required to discuss with the \textbf{LAVA} community, on
\url{irc://irc.freenode.net#linaro-lava}, to get clarification when the
documentation happened to be incomplete, or when \textbf{LAVA} needed to be
improved\footnote{See these patches for example: \\
\url{https://git.linaro.org/lava/lava-dispatcher.git/commit/?id=8df17dd7355cd82f37e1ef22a6c9d88ede44f650} \\
\url{https://git.linaro.org/lava/lava-dispatcher.git/commit/?id=3bfdcdb672f1a15da96bbb221a26847dd6bf2865} \\
Also don't hesitate to run \verb$git log --author "Florent Jacquet"$ in the
\verb$lava-dispatcher$ and the \verb$lava-server$ projects to get an overview of
the contributions made to \textbf{LAVA} (Also available in appendix
\ref{cha:list_of_contributions_to_lava}, page
\pageref{cha:list_of_contributions_to_lava}).
}.

\subsubsection{Using custom artifacts}
\label{ssub:using_custom_artifacts}

Once we could easily send jobs, the next step was about sending custom
artifacts, such as the user-built kernel. This would be useful for the first
manual mode of the tool, when a user would launch some jobs from his
workstation, using a kernel built from one of his working trees.

\textbf{LAVA} allowing the use of files local to the dispatcher, it would be a
really convenient solution to provide the artifacts without setting up some
\textbf{FTP} server or other complicated means of serving files.

\textbf{SSH}, with the \verb$scp$ command, allows efficient and reliable file transfers
between two machines, and since the engineers have an easy access to the
dispatcher using one of Free Electrons' VPNs\footnote{Virtual Private Network
(\url{https://en.wikipedia.org/wiki/Virtual\_private\_network})}, it would be
easy to give them permissions to send files.

With \textbf{Python}, the \textbf{paramiko}\footnote{\url{http://www.paramiko.org/}}
library, allowing a native use of \textbf{SSH}, makes the choice of that
protocol even more comfortable.

\subsubsection{Launching automatic jobs}
\label{ssub:crawling_for_existing_artifacts}

The other mode of the tool, as an automatic launcher, would require to fetch
pre-built artifacts available from a remote location, such as
\textbf{KernelCI}'s storage, or Free Electrons' once the custom builds would be
set up. Fortunately, the \textbf{KernelCI} website also provides an API,
allowing to retrieve their latest builds.

The most difficult part was then to make sure that the crawler would have enough
information about the boards to fetch their specific artifacts, while  trying to
avoid having a very big file storing every possible data about the boards.
Once done, getting the artifacts would only be a matter of crafting the right
URL.

This ended up with a simple \textbf{JSON}\footnote{\url{http://json.org/}}
file, storing the list of the boards, each storing four strings:
\begin{itemize}
    \item \textbf{arch}, the architecture of the board, to guess which kernel to
        use.
    \item \textbf{dt}, the device-tree, also mandatory for booting the devices,
        and unique to each and every one of them.
    \item \textbf{rootfs}, since they are built for many architecture flavours
        (ARMv4, ARMv5, ARMv7, and ARMv8)
    \item \textbf{test\_plan}, since it is mandatory for \textbf{LAVA}, and must
        be configured on a per device basis.
\end{itemize}

This file proved to be simple enough, and the crafter's job is now only about
crafting an URL, and checking if the artifact actually exists.

With the crawlers done, and the rest of the tool already working, the only thing
that remained to be done in \textbf{CTT} was the custom scripts to be run once
the userspace is reached.

\subsection{Running custom scripts}
\label{sub:running_custom_scripts}

\subsubsection{Writing a test suite}
\label{ssub:writing_a_test_suite}

Among the many possibilities brought by the \textbf{LAVA} job structure, is the
possibility of designating a \emph{git} repository and a path in that repo to a
file that would be executed automatically by \textbf{LAVA} from the device's
userland shell.

Before writing more complex tests which would require some time of development,
a simple \verb$echo "Hello world!"$ made just the job. This allowed to do a lot
of testing, checking all possible solutions, and finally define an architecture
that would be both simple and functional enough for the custom tests' needs.

\subsubsection{Integrating custom tools in the root file system}
\label{ssub:integrating_custom_tools_in_the_rootfs}

Before writing more advanced test scripts in the test suite, a problem had to be
solved. Many of the tests would require tools or commands that are not shipped
by default in \textbf{KernelCI}'s rootfs. Moreover, a requirement was that this
rootfs should be compiled for each ARM flavour, unlike the extremely generic one
built by \textbf{KernelCI}.

An easy and flexible way of building custom root filesystems is to use
\textbf{Buildroot}\footnote{\url{https://buildroot.org/}}. This led to some
simple glue scripts\footnote{\url{https://github.com/free-electrons/buildroot-ci}}
building the few configurations requested by the farm, which are mainly
including \emph{iperf}\footnote{\url{https://en.wikipedia.org/wiki/Iperf}
and \url{https://iperf.fr/}} and a full \emph{ping}\footnote{One that includes
the \verb$-f$ option, for ping floods.} version for network stressing, and
\emph{Bonnie++}\footnote{\url{https://en.wikipedia.org/wiki/Bonnie++} and
\url{http://www.coker.com.au/bonnie++/}} for filesystem performances, over a
classic Busybox\footnote{\url{https://en.wikipedia.org/wiki/BusyBox} and
\url{https://busybox.net/}} that provides the rest of the system.

As my first experience with \textbf{Buildroot}, this was a quick but interesting
part that made me discover the power of build systems in the embedded world.

\subsubsection{The road to complex jobs}
\label{ssub:the_road_to_complex_jobs}

With a test suite and custom root filesystem, the overall architecture was in
place. To verify that everything would work as expected, more complex tests were
to be written.

As Busybox provides only \emph{Ash} as its default shell, the scripts needed to
be compatible with this software, and thus could not take advantage of some Bash
features. This turned out to be quite an exercise, since most of the OSs in 2017
provide the latter by default, and the differences may in some cases cause some
headache finding workarounds for complex operations.

The other most interesting part was the development of the first
\emph{Multinode job}\footnote{\url{https://validation.linaro.org/static/docs/v2/multinodeapi.html}}.
This is the \textbf{LAVA} term to describe jobs that require multiple devices,
such as some network-related jobs. Since the boards need to interact, they need
to be synchronized, and \textbf{LAVA} provides some tools in the runtime
environment to allow data exchanges between the devices, but as with classic
threads or processes, this can quickly lead to some race conditions, deadlocks,
or other interesting concurrency problematics.

Once all those problems addressed, with the network tests running, a little
presentation to the team was given, so that everyone would know the status
of the custom Continuous Integration, and this also allowed to show them the
architecture so that they could easily add new boards and tests in the future.

\subsection{Adding some reporting}
\label{sub:adding_some_reporting}

With the two operating modes of \textbf{CTT}, came two modes of reporting: one
for the manual tests, and the other for the daily ones.

For the first and easy part, it was just about adding the correct \emph{notify}
section to the jobs template, so that when an engineer sends a job manually to
\textbf{LAVA}, his email address is included in the definition and he gets a
message as soon as the job is finished, with some details about what worked and
what failed.

For the second part, the daily tests, the need was to aggregate the results of
the past twenty-four hours into a single, personalized email. Indeed, each
engineer can subscribe to some devices, and in order not to make the reporting
too verbose, a script builds a specific email for every one, so that people only
get the results they are interested in.

\subsection{Integrating custom kernels}
\label{sub:integrating_custom_kernels}

\subsubsection{Goal}
\label{ssub:goal}

The next and last step toward fully customized CI tests, was building custom
kernels. Just like \textbf{KernelCI} does every hour, the goal is to monitor a
list of kernel trees, pull them, then build them with specific configurations,
and store the artifacts online, so that \textbf{LAVA} could easily use them.

Custom kernels really come in handy in two cases. When the engineers would like
to follow a specific tree they work on, but this tree is not close enough to
mainline and \textbf{KernelCI} does not track it, Free Electrons' builder would
be in charge of it. The other useful case, is when a test requires custom kernel
configuration, such as the activation of the hardware cryptographic modules,
that are platform specific, thus not in the range of \textbf{KernelCI}'s builds.

\subsubsection{Setting up a kernel builder}
\label{ssub:setting_up_a_kernel_builder}

Mainly based on \textbf{KernelCI}'s Jenkins scripts, but with some
modifications to work in standalone, the builder\footnote{\url{https://github.com/free-electrons/kernel-builder}}
is split in two parts: a first script that checks the trees and prepares
tarballs of the sources when needed, and a second script that builds the
prepared sources.

In the overall CI architecture, the two scripts are called sequentially, just
before launching \textbf{CTT} in automatic mode, so that the newly created
kernel images can quickly be tested. Of course this required adding to
\textbf{CTT} the requested logic to crawl either on \textbf{KernelCI}'s storage,
or Free Electrons' one.

\subsection{A full rework before the end}
\label{sub:a_full_rework_before_the_end}

Before the end of the internship, everything was fully operational, up and
running, but a quite huge problem remained. Indeed, the whole code of
\textbf{CTT} had been developed quickly, as a proof-of-concept, and even if the
technological choices were not bad, the overall design of the software made it
awful to maintain.

The decision was taken, as about one month remained, to rework completely the
tool, so that it would be easier in the future to add new features. The
technical debt brought by the proof-of-concept design pattern would also be
paid.

One of the engineers had already taken time to rework small parts, but was
keeping the internal API untouched when some functions or classes required to be
split in multiple ones. More was needed, but still, he had quite a good vision
of the tool's design, and greatly helped in its refactoring.

This brought along the way many interesting side-effects: unit tests to almost
all the newly created classes, flexible and modular design, simpler
configuration files, better user-interface, improved safety regarding the
crafted jobs, and a fully rewritten README.

Despite not being originally planned in the main subject of the internship, this
truly was an instructive part, since it was all about software design, and
making choices that would help make the tool maintainable for the long term, and
not something that would fall into oblivion in less than six months.