You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

292 lines
11KB

  1. \pdfminorversion=4
  2. \documentclass{beamer}
  3. \usepackage[T1]{ fontenc }
  4. \usepackage[utf8]{ inputenc }
  5. \usepackage{ lmodern }
  6. \usepackage{ color }
  7. \usepackage{graphicx}
  8. \graphicspath{ {./pix/} }
  9. \usepackage{ drawstack }
  10. \usepackage{soul}
  11. \usetheme{metropolis}
  12. \title{Customs tests in Free Electrons's lab}
  13. \subtitle{}
  14. \date{\today}
  15. \author{Florent (Skia) Jacquet}
  16. \institute{Free Electrons}
  17. \begin{document}
  18. \maketitle
  19. \begin{frame}[fragile]\frametitle{Why custom tests}
  20. \begin{itemize}
  21. \item Just booting is not enough.
  22. \item Need to test some specific components such as USB, SATA,
  23. network...
  24. \item Also need to test with custom kernel images, or custom rootfs, or
  25. both.
  26. \item Need to be able to send jobs, on demand, easilly, on the specified
  27. boards, while being notified at the end.
  28. \end{itemize}
  29. \end{frame}
  30. \section{Overall architecture}
  31. \begin{frame}[fragile]\frametitle{Many tools}
  32. \begin{itemize}
  33. \item LAVA
  34. \item Buildroot
  35. \item The custom test tool
  36. \item The test suite
  37. \end{itemize}
  38. \end{frame}
  39. \begin{frame}[fragile]\frametitle{LAVA}
  40. \begin{itemize}
  41. \item \small{\url{https://www.linaro.org/initiatives/lava/}}
  42. \item Runs jobs on devices. \\
  43. A job define a procedure to run, and contains the needed
  44. informations: which kernel image to run on which device, with which
  45. DT/rootfs, and what commands to run in userspace, etc...
  46. \item Runs KernelCI jobs: only boot.
  47. \item Provides an API to do many thing. Among them: sending jobs and
  48. querying results.
  49. \end{itemize}
  50. \end{frame}
  51. \begin{frame}[fragile]{Buildroot}
  52. \begin{itemize}
  53. \item \small{\url{https://github.com/free-electrons/buildroot-ci}}
  54. \item Used to make only rootfs.
  55. \item Is configured to provided the needed tools such as iperf,
  56. bonnie++, etc...
  57. \item Builds fresh rootfs \st{every night} for all the needed architectures:
  58. ARMv4, ARMv5, ARMv7, ARMv8.
  59. \item Stores them in a LAVA-accessible folder in the farm.
  60. \item Small overlay over vanilla buildroot:
  61. \begin{itemize}
  62. \item \verb$./build_rootfs.sh$ to build the rootfs
  63. \item \verb$./configs/ci_*_defconfig$ are the differents
  64. defconfigs.
  65. \item \verb$./board/ci/*$ contains among other small stuff, the
  66. \verb$busybox.config$ file.
  67. \end{itemize}
  68. \end{itemize}
  69. \end{frame}
  70. \begin{frame}[fragile]\frametitle{The custom test tool}
  71. \begin{itemize}
  72. \item \small{\url{https://github.com/free-electrons/custom_tests_tool}}
  73. \item Has job templates and knows how to send them.
  74. \item Has a list of devices in the \verb$boards.py$ file containing the
  75. non-changing data
  76. \item Can fetch latest builds URLs from KernelCI through its API.
  77. \item Can upload custom artifacts useable for tests.
  78. \item Can ask LAVA to notify given emails addresses when the job ends.
  79. \item Needs an access to the LAVA API, an SSH access to the LAVA server
  80. to upload artifacts, and a KernelCI API token to go fetch the latest
  81. builds if no custom artifacts are provided.
  82. \item Keeps a \verb$boards.py$ file to store the boards configuration.
  83. \end{itemize}
  84. \end{frame}
  85. \begin{frame}[fragile]{The test suite}
  86. \begin{itemize}
  87. \item \small{\url{https://github.com/free-electrons/test_suite}}
  88. \item Is a bunch of shell scripts that will be run on the devices.
  89. \item Also needs YAML files to interface with LAVA. \\
  90. This file basically describes a list of commands to be run, and
  91. usually, it only calls the script.
  92. \item In practice, generic YAML files were made for the test suite, but
  93. there may be a need for specific ones in the futur.
  94. \end{itemize}
  95. \begin{scriptsize}
  96. Example:
  97. \label{test_suite_yaml_example}
  98. \begin{verbatim}
  99. metadata:
  100. format: Lava-Test Test Definition 1.0
  101. name: first-test
  102. run:
  103. steps:
  104. - lava-test-case first-test --shell ./scripts/first_test.sh $DEVICE
  105. \end{verbatim}
  106. \end{scriptsize}
  107. \end{frame}
  108. \begin{frame}[fragile]{Overview}
  109. \begin{figure}[H]
  110. \centering
  111. \includegraphics[width=0.8\linewidth]{architecture.png}
  112. \caption{Architecture}
  113. \label{fig:architecture}
  114. \end{figure}
  115. \end{frame}
  116. \begin{frame}[fragile]{Workflow}
  117. Daily jobs:
  118. \begin{itemize}
  119. \item A cron job runs at 23:23 to:
  120. \begin{itemize}
  121. \item \st{Build fresh rootfs.}
  122. \item Launch custom tests on every boards with there
  123. default configuration.
  124. \end{itemize}
  125. \item Another cron is run at 11:00 to get the results of the
  126. last 24 hours and send mails given the failed tests and
  127. which email address is subscribed to what board (in the
  128. \verb$boards.py$ file).
  129. \end{itemize}
  130. \end{frame}
  131. \begin{frame}[fragile]{Workflow (2)}
  132. On demand jobs:
  133. \begin{itemize}
  134. \item \verb$./ctt.py -b all$ \\
  135. \small{sends jobs to all boards with their default tests (it's what
  136. the daily cron job does)}
  137. \item \verb$./ctt.py -b beaglebone-black -m network$ \\
  138. \small{sends only the network test using the multinode job
  139. template (\alert{-m}) to the beaglebone black}
  140. \item \verb$./ctt.py -b armada-7040-db armada-8040-db $ \\
  141. \verb$-t first_test --kernelci-tree next --no-send$ \\
  142. \small{stores the jobs (\alert{$--$no-send}) for the 7040 and 8040
  143. devices, to run only the \alert{first\_test} test on the latest
  144. KernelCI build of the next tree}
  145. \end{itemize}
  146. \end{frame}
  147. \section{How do tests work?}
  148. \begin{frame}[fragile]{LAVA v2 jobs}
  149. \begin{itemize}
  150. \item Jobs are YAML files that, combined with the device-types and
  151. devices dictionaries in the LAVA configuration, describe what to do
  152. with which device.
  153. \item It contains:
  154. \begin{itemize}
  155. \item Which device(s) to use
  156. \item The artifacts informations (URL, type, ...)
  157. \item How to boot them (Uboot, fastboot, ramdisk, nfs, shell
  158. strings to expect, ...)
  159. \item The tests to run once booted.
  160. \item Some metadata and various job informations (priority,
  161. notifications, timeouts, ...)
  162. \end{itemize}
  163. \end{itemize}
  164. \end{frame}
  165. \begin{frame}[fragile]{How do LAVA runs test?}
  166. \begin{itemize}
  167. \item Before bringing up the board, LAVA:
  168. \begin{itemize}
  169. \item Fetches the artifacts.
  170. \item Applies the needed modifications (append DTB, mkimage,
  171. modules, ...)
  172. \item Also downloads the test suite and puts it in the rootfs.
  173. This moment also makes some magic to provide some helpers
  174. useable in the test scripts.
  175. \item Finally connects itself and powers up the board.
  176. \end{itemize}
  177. \item After the board is booted:
  178. \begin{itemize}
  179. \item Runs each test described in the job definition.
  180. \item For each call to the `lava-test-case` command, creates
  181. the corresponding results objects. It's thus important to
  182. call it at least once, or you'll have a job with no results,
  183. apart from the log.
  184. \end{itemize}
  185. \end{itemize}
  186. \end{frame}
  187. \begin{frame}[fragile]{How to write tests?}
  188. \begin{itemize}
  189. \item Tests are in the \verb$test_suite$ repo.
  190. \item You basically just need to write your script in the \verb$scripts$
  191. folder.
  192. \item Don't forget to give it the \verb$.sh$ extension, and for
  193. multinode jobs, they must follow this naming convention:
  194. \verb$testname-laptop.sh$ and \verb$testname-board.sh$.
  195. \item Writing a single test is easy, but it's more complicated to write
  196. MultiNode jobs since you need to synchronize them.
  197. \item In MultiNode, you have access to helper commands
  198. \footnote{Full reference:
  199. \url{https://validation.linaro.org/static/docs/v2/multinodeapi.html}}:
  200. \begin{itemize}
  201. \item \verb$lava-send <message-id> [key1=val1 [key2=val2] ...]$
  202. \item \verb$lava-wait <message-id>$
  203. \end{itemize}
  204. \end{itemize}
  205. \end{frame}
  206. \begin{frame}[fragile]{Full MultiNode example}
  207. Server side:
  208. \begin{scriptsize}
  209. \begin{verbatim}
  210. #!/bin/sh
  211. iperf -s &
  212. echo $! > /tmp/iperf-server.pid
  213. IP=`ip route get 8.8.8.8 | head -n 1 | awk '{print $NF}'`
  214. echo $IP
  215. lava-send server-ready server-ip=$IP
  216. lava-wait client-done
  217. kill -9 `cat /tmp/iperf-server.pid`
  218. \end{verbatim}
  219. \end{scriptsize}
  220. Client side:
  221. \begin{scriptsize}
  222. \begin{verbatim}
  223. #!/bin/sh
  224. lava-wait server-ready
  225. server=$(cat /tmp/lava_multi_node_cache.txt | cut -d = -f 2)
  226. iperf -c $server
  227. # ... do something with output ...
  228. lava-send client-done
  229. \end{verbatim}
  230. \end{scriptsize}
  231. \end{frame}
  232. \begin{frame}[fragile]\frametitle{Collecting the results}
  233. \begin{itemize}
  234. \item LAVA's notification system can send emails if asked. That's the
  235. way results are reported when sending on demand jobs.
  236. \item A daily cron job fetchs the results and aggregates them in an
  237. personalized email to each address in the \verb$boards.py$ file of
  238. the custom test tool.
  239. \item A dashboard is hourly refreshed presenting the devices and the
  240. tests in a table with colors and links to the jobs
  241. \footnote{It's running here: \url{http://farm:5000}}. \\
  242. The code lives in the custom test tool repo, under the
  243. \verb$./dashboard$ folder.
  244. \end{itemize}
  245. \end{frame}
  246. \begin{frame}[fragile]{Sum up, improving, contributing...}
  247. The READMEs should already explain much of what you'll need.
  248. \begin{itemize}
  249. \item To add more tools to the rootfs: \\
  250. \small{\url{https://github.com/free-electrons/buildroot-ci}}
  251. \item To add more boards once they are in LAVA, or add more tests to a
  252. board, or to modify the job templates, or to improve the dashboard,
  253. or simply to subscribe to a device: \\
  254. \small{\url{https://github.com/free-electrons/custom_tests_tool}}
  255. \item To add more tests in the test suite: \\
  256. \small{\url{https://github.com/free-electrons/test_suite}}
  257. \end{itemize}
  258. \end{frame}
  259. \begin{frame}[fragile]\frametitle{Questions}
  260. \begin{center}
  261. \alert{Question?}
  262. \end{center}
  263. \end{frame}
  264. \end{document}