You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

289 line
11KB

  1. \pdfminorversion=4
  2. \documentclass{beamer}
  3. \usepackage[T1]{ fontenc }
  4. \usepackage[utf8]{ inputenc }
  5. \usepackage{ lmodern }
  6. \usepackage{ color }
  7. \usepackage{ graphicx }
  8. \graphicspath{ {./pix/} }
  9. \usepackage{ drawstack }
  10. \usetheme{metropolis}
  11. \title{Customs tests in Free Electrons's lab}
  12. \subtitle{}
  13. \date{\today}
  14. \author{Florent (Skia) Jacquet}
  15. \institute{Free Electrons}
  16. \begin{document}
  17. \maketitle
  18. \begin{frame}[fragile]\frametitle{Why custom tests}
  19. \begin{itemize}
  20. \item Just booting is not enough.
  21. \item Need to test some specific components such as USB, SATA,
  22. network...
  23. \item Also need to test with custom kernel images, or custom rootfs, or
  24. both.
  25. \item Need to be able to send jobs, on demand, easilly, on the specified
  26. boards, while being notified at the end.
  27. \end{itemize}
  28. \end{frame}
  29. \section{Overall architecture}
  30. \begin{frame}[fragile]\frametitle{Many tools}
  31. \begin{itemize}
  32. \item LAVA
  33. \item Buildroot
  34. \item The custom test tool
  35. \item The test suite
  36. \end{itemize}
  37. \end{frame}
  38. \begin{frame}[fragile]\frametitle{LAVA}
  39. \begin{itemize}
  40. \item \small{\url{https://www.linaro.org/initiatives/lava/}}
  41. \item Runs jobs on devices. \\
  42. A job define a procedure to run, and contains the needed
  43. informations: which kernel image to run on which device, with which
  44. DT/rootfs, and what commands to run in userspace, etc...
  45. \item Runs KernelCI jobs: only boot.
  46. \item Provides an API to do many thing. Among them: sending jobs and
  47. querying results.
  48. \end{itemize}
  49. \end{frame}
  50. \begin{frame}[fragile]{Buildroot}
  51. \begin{itemize}
  52. \item \small{\url{https://github.com/free-electrons/buildroot-ci}}
  53. \item Used to make only rootfs.
  54. \item Is configured to provided the needed tools such as iperf,
  55. bonnie++, etc...
  56. \item Builds fresh rootfs every night for all the needed architectures:
  57. ARMv4, ARMv5, ARMv7, ARMv8.
  58. \item Stores them in a LAVA-accessible folder in the farm.
  59. \item Small overlay over vanilla buildroot:
  60. \begin{itemize}
  61. \item \verb$./build_rootfs.sh$ to build the rootfs
  62. \item \verb$./configs/ci_*_defconfig$ are the differents
  63. defconfigs.
  64. \item \verb$./board/ci/*$ contains among other small stuff, the
  65. \verb$busybox.config$ file.
  66. \end{itemize}
  67. \end{itemize}
  68. \end{frame}
  69. \begin{frame}[fragile]\frametitle{The custom test tool}
  70. \begin{itemize}
  71. \item \small{\url{https://github.com/free-electrons/custom_tests_tool}}
  72. \item Has job templates and knows how to send them.
  73. \item Has a list of devices in the \verb$boards.py$ file containing the
  74. non-changing data
  75. \item Can fetch latest builds URLs from KernelCI.
  76. \item Can upload custom artifacts useable for tests.
  77. \item Can ask LAVA to notify given emails addresses when the job ends.
  78. \item Needs an access to the LAVA API, an SSH access to the LAVA server
  79. to upload artifacts, and a KernelCI API token to go fetch the latest
  80. builds if no custom artifacts are provided.
  81. \item Keeps a \verb$boards.py$ file to store the boards configuration.
  82. \end{itemize}
  83. \end{frame}
  84. \begin{frame}[fragile]{The test suite}
  85. \begin{itemize}
  86. \item \small{\url{https://github.com/free-electrons/test_suite}}
  87. \item Is a bunch of shell scripts that will be run on the devices.
  88. \item Also needs YAML files to interface with LAVA. \\
  89. This file basically describes a list of commands to be run, and
  90. usually, it only calls the script.
  91. \end{itemize}
  92. \begin{scriptsize}
  93. Example:
  94. \label{test_suite_yaml_example}
  95. \begin{verbatim}
  96. metadata:
  97. format: Lava-Test Test Definition 1.0
  98. name: first-test
  99. run:
  100. steps:
  101. - lava-test-case first-test --shell ./scripts/first_test.sh $DEVICE
  102. \end{verbatim}
  103. \end{scriptsize}
  104. \end{frame}
  105. \begin{frame}[fragile]{Overview}
  106. \begin{figure}[H]
  107. \centering
  108. \includegraphics[width=0.8\linewidth]{architecture.png}
  109. \caption{Architecture}
  110. \label{fig:architecture}
  111. \end{figure}
  112. \end{frame}
  113. \begin{frame}[fragile]{Workflow}
  114. Daily jobs:
  115. \begin{itemize}
  116. \item A cron job runs at 23:23 to:
  117. \begin{itemize}
  118. \item Build fresh rootfs.
  119. \item Launch custom tests on every boards with there
  120. default configuration.
  121. \end{itemize}
  122. \item Another cron is run at 11:00 to get the results of the
  123. last 24 hours and send mails given the failed tests and
  124. which email address is subscribed to what board (in the
  125. \verb$boards.py$ file).
  126. \end{itemize}
  127. \end{frame}
  128. \begin{frame}[fragile]{Workflow (2)}
  129. On demand jobs:
  130. \begin{itemize}
  131. \item \verb$./ctt.py -b all --send$ \\
  132. \small{sends jobs to all boards with their default tests (it's what
  133. the daily cron job does)}
  134. \item \verb$./ctt.py -b beaglebone-black -m network --send$ \\
  135. \small{sends only the network test using the multinode job
  136. template (\alert{-m}) to the beaglebone black}
  137. \item \verb$./ctt.py -b armada-7040-db armada-8040-db $ \\
  138. \verb$-t first_test --kernelci-tree next$ \\
  139. \small{stores the jobs (no \alert{$--$send}) for the 7040 and 8040
  140. devices, to run only the \alert{first\_test} test on the latest
  141. KernelCI build of the next tree}
  142. \end{itemize}
  143. \end{frame}
  144. \section{How do tests works?}
  145. \begin{frame}[fragile]{LAVA v2 jobs}
  146. \begin{itemize}
  147. \item Jobs are YAML files that, combined with the device-types and
  148. devices dictionaries in the LAVA configuration, describe what to do
  149. with which device.
  150. \item It contains:
  151. \begin{itemize}
  152. \item Which device(s) to use
  153. \item The artifacts informations (URL, type, ...)
  154. \item How to boot them (Uboot, fastboot, ramdisk, nfs, shell
  155. strings to expect, ...)
  156. \item The tests to run once booted.
  157. \item Some metadata and various job informations (priority,
  158. notifications, timeouts, ...)
  159. \end{itemize}
  160. \end{itemize}
  161. \end{frame}
  162. \begin{frame}[fragile]{How do LAVA runs test?}
  163. \begin{itemize}
  164. \item Before bringing up the board, LAVA:
  165. \begin{itemize}
  166. \item Fetches the artifacts.
  167. \item Applies the needed modifications (append DTB, mkimage,
  168. modules, ...)
  169. \item Also downloads the test suite and puts it in the rootfs.
  170. This moment also makes some magic to provide some helpers
  171. useable in the test scripts.
  172. \item Finally connects itself and powers up the board.
  173. \end{itemize}
  174. \item After the board is booted:
  175. \begin{itemize}
  176. \item Runs each test described in the job definition.
  177. \item For each call to the `lava-test-case` command, creates
  178. the corresponding results objects. It's thus important to
  179. call it at least once, or you'll have a job with no results,
  180. apart from the log.
  181. \end{itemize}
  182. \end{itemize}
  183. \end{frame}
  184. \begin{frame}[fragile]{How to write tests?}
  185. \begin{itemize}
  186. \item Tests are in the \verb$test_suite$ repo.
  187. \item You basically just need to write your script in the \verb$scripts$
  188. folder.
  189. \item Don't forget to give it the \verb$.sh$ extension, and for
  190. multinode jobs, they must follow this naming convention:
  191. \verb$testname-laptop.sh$ and \verb$testname-board.sh$.
  192. \item Writing a single test is easy, but it's more complicated to write
  193. MultiNode jobs since you need to synchronize them.
  194. \item In MultiNode, you have access to helper commands
  195. \footnote{Full reference:
  196. \url{https://validation.linaro.org/static/docs/v2/multinodeapi.html}}:
  197. \begin{itemize}
  198. \item \verb$lava-send <message-id> [key1=val1 [key2=val2] ...]$
  199. \item \verb$lava-wait <message-id>$
  200. \end{itemize}
  201. \end{itemize}
  202. \end{frame}
  203. \begin{frame}[fragile]{Full MultiNode example}
  204. Server side:
  205. \begin{scriptsize}
  206. \begin{verbatim}
  207. #!/bin/sh
  208. iperf -s &
  209. echo $! > /tmp/iperf-server.pid
  210. IP=`ip route get 8.8.8.8 | head -n 1 | awk '{print $NF}'`
  211. echo $IP
  212. lava-send server-ready server-ip=$IP
  213. lava-wait client-done
  214. kill -9 `cat /tmp/iperf-server.pid`
  215. \end{verbatim}
  216. \end{scriptsize}
  217. Client side:
  218. \begin{scriptsize}
  219. \begin{verbatim}
  220. #!/bin/sh
  221. lava-wait server-ready
  222. server=$(cat /tmp/lava_multi_node_cache.txt | cut -d = -f 2)
  223. iperf -c $server
  224. # ... do something with output ...
  225. lava-send client-done
  226. \end{verbatim}
  227. \end{scriptsize}
  228. \end{frame}
  229. \begin{frame}[fragile]\frametitle{Collecting the results}
  230. \begin{itemize}
  231. \item LAVA's notification system can send emails if asked. That's the
  232. way results are reported when sending on demand jobs.
  233. \item A daily cron job fetchs the results and aggregates them in an
  234. personalized email to each address in the \verb$boards.py$ file of
  235. the custom test tool.
  236. \item A dashboard is hourly refreshed presenting the devices and the
  237. tests in a table with colors and links to the jobs
  238. \footnote{It's running here: \url{http://farm:5000}}. \\
  239. The code lives in the custom test tool repo, under the
  240. \verb$./dashboard$ folder.
  241. \end{itemize}
  242. \end{frame}
  243. \begin{frame}[fragile]{Sum up, improving, contributing...}
  244. The READMEs should already explain much of what you'll need.
  245. \begin{itemize}
  246. \item To add more tools to the rootfs: \\
  247. \small{\url{https://github.com/free-electrons/buildroot-ci}}
  248. \item To add more boards once they are in LAVA, or add more tests to a
  249. board, or to modify the job templates, or to improve the dashboard,
  250. or simply to subscribe to a device: \\
  251. \small{\url{https://github.com/free-electrons/custom_tests_tool}}
  252. \item To add more tests in the test suite: \\
  253. \small{\url{https://github.com/free-electrons/test_suite}}
  254. \end{itemize}
  255. \end{frame}
  256. \begin{frame}[fragile]\frametitle{Questions}
  257. \begin{center}
  258. \alert{Question?}
  259. \end{center}
  260. \end{frame}
  261. \end{document}