You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

489 lines 24KB Raw Permalink Blame History

 \chapter{Custom tests for Continuous Integration} \label{cha:custom_tests_for_continuous_integration} \emph{First of all, it is important to note the distinction between the three following elements of the architecture: KernelCI, the custom tests, and the LAVA lab. All of them will be detailed in this chapter, but they are all completely independent, and are simply working together in Free Electrons' CI infrastructure.} \section{The need for custom tests} \label{sec:the_need_for_custom_tests} \subsection{The KernelCI project} \label{sub:the_kernelci_project} \textbf{KernelCI} is a project started a few years ago that aims at compiling on a per hour basis a lot of different upstream \textbf{Linux} kernel trees, then sending jobs\footnote{Jobs are the resource \textbf{LAVA} deals with. A more detailed explanation can be found in \ref{ssub:the_jobs} at page \pageref{ssub:the_jobs}.} using those kernels to the subscribed labs, before aggregating the results on different summary pages, where it is easy to see if a board has some problems booting \textbf{Linux}, and when the problems started to occur. This project already does a great part of the CI loop by building and displaying the results, but still needs the collaboration of the many labs contributing their device availability to have the complete process. With all the different kind of devices provided by the labs from all over the world, they have achieved over the years a quite good coverage of the supported device types in \textbf{Linux}. But even if they test a great number of platforms, the only jobs they send make little more than booting and running a few basic commands in userspace. Jobs as simple as this are not suitable when it comes to test the SATA, the USB, or some other specific systems that are usually unused during boot. But of course, as \textbf{KernelCI} has to deal with many different labs, they can not afford to make the jobs more specific, since many boards, despite being identical, may have different configurations and different devices attached to it. To fill that gap, custom tests must be set up at a smaller scale, to comply with the specificities of the lab and the devices. \subsection{Specifications} \label{sub:specifications} As Free Electrons counts many device family maintainers in its engineering team, it is particularly important for them to have the finest CI set up for those devices. Moreover, they frequently work and develop new drivers on custom \textbf{Linux} trees, often coming from the device's vendor tree. Since this process can take quite a long time, it would be wonderful to have some CI process in place for those custom trees, and custom features still being in development. Considering the jobs already run by \textbf{KernelCI}, something more specific but still mostly similar was needed to implement the custom tests\footnote{The term \emph{custom tests} designates tests sent by a different system than \textbf{KernelCI}'s, that would be fully controlled by \textbf{Free Electrons}, and could thus be tuned finely for the engineers needs.}. Two parts were to be set up: \begin{itemize} \item launching custom scripts on the devices to check the specific functionalities. \item building and booting custom kernel trees that are not already taken care of by \textbf{KernelCI}. \end{itemize} Of course, once both parts are running, they can be combined to launch custom scripts on custom kernels. Last but not least, the overall architecture should be supporting two operating modes: \begin{itemize} \item a \textbf{manual} one, that could be triggered by hand, to run only the asked specific tests with a user-built kernel, and give immediate report once the job has been run. \item an \textbf{automatic} one, that would run tests every day, and make daily reports to the maintainers. \end{itemize} \section{The CI lab} \label{sec:the_ci_lab} Applying continuous integration to an operating system is not an easy task. Since it needs external hardware management, power supply control, and input/output control, it clearly requires a complex infrastructure. Free Electrons has a lab now for more than one year, that it is running fine, continuously testing more than 35 devices, and reporting public results making them available for the \textbf{Linux} community. \subsection{The hardware part} \label{sub:the_hardware_part} Since the tested software is an operating system, it needs to run on real hardware, and thus, it differs from more usual CI that typically runs in some container technology. Built last year, the farm at Free Electrons takes the form of a big cabinet, with eight drawers, capable of storing up to 50 boards. With those devices, USB hubs, switches, and ATX power supply with their TCP controlled relays can be found in each and every drawer. For the main management, the farm also hosts a main USB hub, a main switch, and most importantly, the NUC that runs parts of the software driving the whole lab. Everything is well wired and labelled to keep the material in a maintainable state. For more information about the ins and outs of this part, one can read this blog article: \url{http://free-electrons.com/blog/hardware-infrastructure-free-electrons-lab/}. \begin{figure}[H] \centering \includegraphics[height=0.4\paperheight]{lab.JPG} \caption{An overview of the cabinet hosting the hardware part of the lab.} \label{fig:lab} \end{figure} \subsection{The software: LAVA} \label{sub:the_lava_software} In order to run everything from a software point of view, \textbf{LAVA} has been used from the very beginning. LAVA stands for \emph{Linaro Automated Validation Architecture}, and provides a quite good way to manage and schedule automated tests on embedded devices. Most of the job to do to implement custom tests was about using \textbf{LAVA} the right way to perform the tests, so this part will include more details than the hardware one. \subsubsection{LAVA: v1 or v2?} \label{ssub:lava_v1_or_v2} Since \textbf{LAVA} is currently in the middle of a huge refactoring of its internal workflow and exposed API, we usually find people talking about \textbf{LAVA v1} and \textbf{LAVA v2}. It is actually the same software, with different behaviors at many levels, but with the same final goal: running what is called a job, a user described list of actions, on a device, and reporting the events that happened. At the beginning of the internship, only the first version was in use, and part of the job was about migrating to the second one. This ran through a lot of different problems and bugs that needed to be fixed before taking the next step, but we finally ended up with a fresh and running architecture using mostly \textbf{LAVA v2}. For clarity's sake, only the \textbf{final setup} will be described in this report. Those interested in the original software architecture can still read this blog post: \url{http://free-electrons.com/blog/software-architecture-free-electrons-lab/}. \subsubsection{The jobs} \label{ssub:the_jobs} The main resource \textbf{LAVA} has to deal with is the job. A job is defined by a \textbf{YAML}\footnote{\url{http://yaml.org/}} structure, describing multiple sections: \begin{itemize} \item The \textbf{device-type}, which is the name of the device you want to run the test on. \\ \emph{Examples: beaglebone-black, sun8i-h3-orangepi-pc, ...} \item The \textbf{boot method}, to tell \textbf{LAVA} which method should be used to boot the device. \\ \emph{Examples: \textbf{fastboot} or \textbf{U-boot}, \textbf{NFS} or ramdisk, \textbf{SSH}, ...} \item The \textbf{artifacts} URLs. This includes the \emph{kernel}, the \emph{device tree}, the \emph{modules}, and the \emph{rootfs}. Only the kernel is completely mandatory to boot the boards, but the other ones are common for almost every non-exotic device. \item The \textbf{tests} to run once the device is booted. This can include many possibilities, since it generally points to some shell scripts to be executed as root in userspace. It is the main place to customize the tests. \item Other less important sections, such as some \textbf{metadata}, the \textbf{notifications}, or custom \textbf{timeouts}. \end{itemize} Once a job has been sent either by the web interface or by the API, it is queued until a device corresponding to the asked device-type is free to be used. Then the job gets scheduled and run, before finishing either with the \textbf{Complete} status when everything ran well, or the \textbf{Incomplete} status when there was a problem during the execution of the different tasks. When a test is complete, \textbf{LAVA} provides access to the results by many ways, such as the web UI, the API, some emails, or a callback system making the job able to push its results to some other APIs. \subsubsection{A distributed system} \label{ssub:a_distributed_system} \textbf{LAVA} has been designed to scale for far more complex labs than Free Electrons' one. It is thus split into two parts: a master, and a worker. \begin{description} \item[The master]: \\ The master can exist in only instance. It is in charge of three tasks: \begin{itemize} \item The \textbf{web interface}, allowing in-browser interaction with the software. \item The job \textbf{scheduler}, responsible for sending the queued jobs to the available devices. \item The \textbf{dispatcher-master}, that manages the different possible workers, and sends jobs to them. \end{itemize} The master also has the connection to the relational database. \item[The worker]: \\ The worker is divided in only two parts: \begin{itemize} \item The \textbf{slave} is the part that connects to the dispatcher-master and receives the jobs to run. \item The \textbf{dispatcher} is the only part that really interacts with the devices under test. It is spawned on demand by the slave when a job needs to be run. It is also the only part that does not run as a daemon. \end{itemize} \end{description} \begin{figure}[H] \centering \includegraphics[width=0.8\linewidth]{arch-overview.png} \caption{Architecture schematic of the \textbf{LAVA} software.} \label{fig:arch-overview} \end{figure} \section{Developing custom tests} \label{sec:developing_custom_tests} \subsection{Beginning a proof of concept} \label{sub:beginning_a_proof_of_concept} At the beginning, a basic and functional, but still blurred specification was made, but it required a proof-of-concept to see how it would fit in final production. It had quickly been named \textbf{CTT}, standing for \emph{Custom Test Tool}\footnote{You can find the sources at this address: \url{https://github.com/free-electrons/custom_tests_tool}}, and that is how the software building the custom jobs will be designated till the end of this report. The choice of \textbf{Python} was more than obvious, since this language is accessible and widely used in the embedded world for its flexibility and portability. Moreover, most of the Free Electrons engineers had already used it, and it was not an option to introduce a new, unknown technology, in an architecture they would have to maintain in the future. \subsubsection{Understanding the depth of LAVA's jobs} \label{ssub:understanding_the_lava_jobs} The first simple part was about \textbf{LAVA}. Since \textbf{KernelCI} already provides everything (kernel, dtb and rootfs) needed to run a successful job in \textbf{LAVA}, the only part remaining was crafting and sending jobs. An easy and simple, yet flexible solution, was to use a template engine to parametrize a generic job written once and for all. The job syntax is using the human friendly \textbf{YAML}, but even if it is readable and easy to write, the data structure itself required by \textbf{LAVA} is a bit complex, and it is thus truly inconvenient to write tests by hand. Once filled, the template would just have to be sent to \textbf{LAVA} through its XML-RPC\footnote{\url{https://en.wikipedia.org/wiki/XML-RPC}} API to create a job. Knowing what to put in that template was one of the most interesting moment of this part, since it was like discovering a new programming language. There are always new features to discover, and new mechanisms for using them, and finally to make \textbf{LAVA} do exactly what you want. It is also during this period that most of the migration to \textbf{LAVA v2} was prepared, meaning that the configuration of the different levels of \textbf{LAVA} was altered. It was often required to discuss with the \textbf{LAVA} community, on \url{irc://irc.freenode.net#linaro-lava}, to get clarification when the documentation happened to be incomplete, or when \textbf{LAVA} needed to be improved\footnote{See these patches for example: \\ \url{https://git.linaro.org/lava/lava-dispatcher.git/commit/?id=8df17dd7355cd82f37e1ef22a6c9d88ede44f650} \\ \url{https://git.linaro.org/lava/lava-dispatcher.git/commit/?id=3bfdcdb672f1a15da96bbb221a26847dd6bf2865} \\ Also don't hesitate to run \verb$git log --author "Florent Jacquet"$ in the \verb$lava-dispatcher$ and the \verb$lava-server$ projects to get an overview of the contributions made to \textbf{LAVA} (Also available in appendix \ref{cha:list_of_contributions_to_lava}, page \pageref{cha:list_of_contributions_to_lava}). }. \subsubsection{Using custom artifacts} \label{ssub:using_custom_artifacts} Once we could easily send jobs, the next step was about sending custom artifacts, such as the user-built kernel. This would be useful for the first manual mode of the tool, when a user would launch some jobs from his workstation, using a kernel built from one of his working trees. \textbf{LAVA} allowing the use of files local to the dispatcher, it would be a really convenient solution to provide the artifacts without setting up some \textbf{FTP} server or other complicated means of serving files. \textbf{SSH}, with the \verb$scp$ command, allows efficient and reliable file transfers between two machines, and since the engineers have an easy access to the dispatcher using one of Free Electrons' VPNs\footnote{Virtual Private Network (\url{https://en.wikipedia.org/wiki/Virtual\_private\_network})}, it would be easy to give them permissions to send files. With \textbf{Python}, the \textbf{paramiko}\footnote{\url{http://www.paramiko.org/}} library, allowing a native use of \textbf{SSH}, makes the choice of that protocol even more comfortable. \subsubsection{Launching automatic jobs} \label{ssub:crawling_for_existing_artifacts} The other mode of the tool, as an automatic launcher, would require to fetch pre-built artifacts available from a remote location, such as \textbf{KernelCI}'s storage, or Free Electrons' once the custom builds would be set up. Fortunately, the \textbf{KernelCI} website also provides an API, allowing to retrieve their latest builds. The most difficult part was then to make sure that the crawler would have enough information about the boards to fetch their specific artifacts, while trying to avoid having a very big file storing every possible data about the boards. Once done, getting the artifacts would only be a matter of crafting the right URL. This ended up with a simple \textbf{JSON}\footnote{\url{http://json.org/}} file, storing the list of the boards, each storing four strings: \begin{itemize} \item \textbf{arch}, the architecture of the board, to guess which kernel to use. \item \textbf{dt}, the device-tree, also mandatory for booting the devices, and unique to each and every one of them. \item \textbf{rootfs}, since they are built for many architecture flavours (ARMv4, ARMv5, ARMv7, and ARMv8) \item \textbf{test\_plan}, since it is mandatory for \textbf{LAVA}, and must be configured on a per device basis. \end{itemize} This file proved to be simple enough, and the crafter's job is now only about crafting an URL, and checking if the artifact actually exists. With the crawlers done, and the rest of the tool already working, the only thing that remained to be done in \textbf{CTT} was the custom scripts to be run once the userspace is reached. \subsection{Running custom scripts} \label{sub:running_custom_scripts} \subsubsection{Writing a test suite} \label{ssub:writing_a_test_suite} Among the many possibilities brought by the \textbf{LAVA} job structure, is the possibility of designating a \emph{git} repository and a path in that repo to a file that would be executed automatically by \textbf{LAVA} from the device's userland shell. Before writing more complex tests which would require some time of development, a simple \verb$echo "Hello world!"$ made just the job. This allowed to do a lot of testing, checking all possible solutions, and finally define an architecture that would be both simple and functional enough for the custom tests' needs. \subsubsection{Integrating custom tools in the root file system} \label{ssub:integrating_custom_tools_in_the_rootfs} Before writing more advanced test scripts in the test suite, a problem had to be solved. Many of the tests would require tools or commands that are not shipped by default in \textbf{KernelCI}'s rootfs. Moreover, a requirement was that this rootfs should be compiled for each ARM flavour, unlike the extremely generic one built by \textbf{KernelCI}. An easy and flexible way of building custom root filesystems is to use \textbf{Buildroot}\footnote{\url{https://buildroot.org/}}. This led to some simple glue scripts\footnote{\url{https://github.com/free-electrons/buildroot-ci}} building the few configurations requested by the farm, which are mainly including \emph{iperf}\footnote{\url{https://en.wikipedia.org/wiki/Iperf} and \url{https://iperf.fr/}} and a full \emph{ping}\footnote{One that includes the \verb$-f$ option, for ping floods.} version for network stressing, and \emph{Bonnie++}\footnote{\url{https://en.wikipedia.org/wiki/Bonnie++} and \url{http://www.coker.com.au/bonnie++/}} for filesystem performances, over a classic Busybox\footnote{\url{https://en.wikipedia.org/wiki/BusyBox} and \url{https://busybox.net/}} that provides the rest of the system. As my first experience with \textbf{Buildroot}, this was a quick but interesting part that made me discover the power of build systems in the embedded world. \subsubsection{The road to complex jobs} \label{ssub:the_road_to_complex_jobs} With a test suite and custom root filesystem, the overall architecture was in place. To verify that everything would work as expected, more complex tests were to be written. As Busybox provides only \emph{Ash} as its default shell, the scripts needed to be compatible with this software, and thus could not take advantage of some Bash features. This turned out to be quite an exercise, since most of the OSs in 2017 provide the latter by default, and the differences may in some cases cause some headache finding workarounds for complex operations. The other most interesting part was the development of the first \emph{Multinode job}\footnote{\url{https://validation.linaro.org/static/docs/v2/multinodeapi.html}}. This is the \textbf{LAVA} term to describe jobs that require multiple devices, such as some network-related jobs. Since the boards need to interact, they need to be synchronized, and \textbf{LAVA} provides some tools in the runtime environment to allow data exchanges between the devices, but as with classic threads or processes, this can quickly lead to some race conditions, deadlocks, or other interesting concurrency problematics. Once all those problems addressed, with the network tests running, a little presentation to the team was given, so that everyone would know the status of the custom Continuous Integration, and this also allowed to show them the architecture so that they could easily add new boards and tests in the future. \subsection{Adding some reporting} \label{sub:adding_some_reporting} With the two operating modes of \textbf{CTT}, came two modes of reporting: one for the manual tests, and the other for the daily ones. For the first and easy part, it was just about adding the correct \emph{notify} section to the jobs template, so that when an engineer sends a job manually to \textbf{LAVA}, his email address is included in the definition and he gets a message as soon as the job is finished, with some details about what worked and what failed. For the second part, the daily tests, the need was to aggregate the results of the past twenty-four hours into a single, personalized email. Indeed, each engineer can subscribe to some devices, and in order not to make the reporting too verbose, a script builds a specific email for every one, so that people only get the results they are interested in. \subsection{Integrating custom kernels} \label{sub:integrating_custom_kernels} \subsubsection{Goal} \label{ssub:goal} The next and last step toward fully customized CI tests, was building custom kernels. Just like \textbf{KernelCI} does every hour, the goal is to monitor a list of kernel trees, pull them, then build them with specific configurations, and store the artifacts online, so that \textbf{LAVA} could easily use them. Custom kernels really come in handy in two cases. When the engineers would like to follow a specific tree they work on, but this tree is not close enough to mainline and \textbf{KernelCI} does not track it, Free Electrons' builder would be in charge of it. The other useful case, is when a test requires custom kernel configuration, such as the activation of the hardware cryptographic modules, that are platform specific, thus not in the range of \textbf{KernelCI}'s builds. \subsubsection{Setting up a kernel builder} \label{ssub:setting_up_a_kernel_builder} Mainly based on \textbf{KernelCI}'s Jenkins scripts, but with some modifications to work in standalone, the builder\footnote{\url{https://github.com/free-electrons/kernel-builder}} is split in two parts: a first script that checks the trees and prepares tarballs of the sources when needed, and a second script that builds the prepared sources. In the overall CI architecture, the two scripts are called sequentially, just before launching \textbf{CTT} in automatic mode, so that the newly created kernel images can quickly be tested. Of course this required adding to \textbf{CTT} the requested logic to crawl either on \textbf{KernelCI}'s storage, or Free Electrons' one. \subsection{A full rework before the end} \label{sub:a_full_rework_before_the_end} Before the end of the internship, everything was fully operational, up and running, but a quite huge problem remained. Indeed, the whole code of \textbf{CTT} had been developed quickly, as a proof-of-concept, and even if the technological choices were not bad, the overall design of the software made it awful to maintain. The decision was taken, as about one month remained, to rework completely the tool, so that it would be easier in the future to add new features. The technical debt brought by the proof-of-concept design pattern would also be paid. One of the engineers had already taken time to rework small parts, but was keeping the internal API untouched when some functions or classes required to be split in multiple ones. More was needed, but still, he had quite a good vision of the tool's design, and greatly helped in its refactoring. This brought along the way many interesting side-effects: unit tests to almost all the newly created classes, flexible and modular design, simpler configuration files, better user-interface, improved safety regarding the crafted jobs, and a fully rewritten README. Despite not being originally planned in the main subject of the internship, this truly was an instructive part, since it was all about software design, and making choices that would help make the tool maintainable for the long term, and not something that would fall into oblivion in less than six months.