In this post, you'll get acquainted with the Yocto Project. A very powerful set of tools for creating and building complete Linux Systems, in particular, embedded systems. After a quick discussing on the tool itself, there's a tutorial on how to use Yocto to build the Angstrom Distribution completely from source to work on a Raspberry Pi 3. Head down to go directly to the tutorial!
The Yocto Project
I think the best way to explain what Yocto is, is to show what it is trying to solve. Generating Linux systems, specially embedded ones, can be astronomically complex if you do it manually. You can always try to find online a prepackaged distro and see if it fits your needs. If you're working with embedded systems, it's most likely that you will not find a suitable candidate.
Embedded systems are very particular and, normally, very resource limited. You need to put exactly what it's need in there; nothing more. So, how can build yourself a custom Linux Distro without having to set sail in a titanic journey that you're most likely to give up in the middle. Enter the Yocto Project!
The Yocto Project (YP) is an open source collaboration project that helps developers create custom Linux-based systems regardless of the hardware architecture. - Yocto Project
Yocto is a comprehensive set of tools that help you automate the building of a complete Linux system from source. It cross-compiles the kernel, modules, apps; creates the rootfs; packages the image; and even prepares tool chains enabling you to develop software for your brand new system.
YP is really powerful but there's a caveat: the learning curve for it is steep, specially when you try to understand what's going on under the hood. It is steep but not as steep as learning how to do every single step manually. The thing is that Yocto's customization is virtually endless. Think of it as an automation tool. Feed it with recipes and it will execute them for you. These recipes can be very simple - like describing how to create a particular directory tree and populate them with random text files - or very complex - like compiling the Linux kernel for the target architecture. They can depend on other recipes and even be components of them. Begin chaining these recipes together and when you realize, you'll have yourself a full fledged Distro.
Another selling point on Yocto is that it is an open source collaboration project. This means that, probably, some good soul on the Internet has already made a recipe for the package you're looking for. Recipes are organized in packages called Layers that you'll stack up to build your environment. This will get more clear in a moment. If you head up to the OpenEmbedded Layer Index you will find a library of layers and their git links. Notice that the layers are divided into the following categories: Base, Machine(BSP), Distribution, Miscellaneous and Software. Each one addresses a type of component of the final system.
In a grossly simplified example, let's say you have an Allwinner based embedded computer and you need to run a nodejs app in it. You could stack the following layers among others:
meta-sunxi - Machine(BSP) - Allwinner board support
meta-poky - Distro - Yocto reference Distro
meta-nodejs - Software - App Framework
I'm omitting lots of steps in between but I guess you get the picture. After building the system, you'll have a complete Distro compiled and prepared to boot on this Allwinner board with exactly the software you needed to run your application. If you fell like learning more, I encourage you to read more about Yocto and its build engine OpenEmbedded. Be prepared, the documentation is extensive.
Let's try it
The best way to learn Yocto, is by getting your hands dirty. I guess the following example is a good way to start. A Yocto "Hello World" if you will.
We'll build the Angstrom Linux Distro for a Raspberry Pi 3. Angstrom is a very reduced Distro; it will pretty much get you a blinking cursor at the end. I mean it, the complete system takes no more the 58mb. Think of it as a good place to base your future Distro from; since there is really no point on starting from nothing. Now, creating your own Angstrom based Distro is something for another time.
Today we will just get Angstrom going as it is. Actually, let's make a single modification. Let's use U-Boot as our bootloader. We'll do this by changing a single line of code. By doing so, Yocto will compile a different bootloader, install it and set up a completely different boot sequence with the addition of a single line. Our course, the recipe for this process is longer than a line of code but it is amazing how you can activate it and encapsulate the whole process in a line of code. This shows the power of customization on Yocto.
Compiling a distro is very resource intensive. Be patient, depending on the host system, the process can take many hours. In addition, it can take up to 20Gb of storage space so make sure you have at least that free on your workspace.
1. First, get your host machine ready.
Ubuntu and Debian
$ sudo apt-get install gawk wget git-core diffstat unzip texinfo gcc-multilib build-essential chrpath socat libsdl1.2-dev xterm Fedora
$ sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl patch diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat findutils which SDL-devel xterm OpenSUSE
$ sudo zypper install python gcc gcc-c++ git chrpath make wget python-xml diffstat makeinfo python-curses patch socat libSDL-devel xterm CentOS
$ sudo yum install gawk make wget tar bzip2 gzip python unzip perl patch diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath socat perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue SDL-devel xterm 2. Now lets clone Yocto. At the time of writing, the latest version o Yocto is code named thud; this will be the version used.
$ git clone -b thud http://git.yoctoproject.org/git/poky yocto
$ git clone -b thud git://git.openembedded.org/meta-openembedded yocto/meta-openembedded3. Clone the Angstrom Distro layer. Notice all of the following branches are also named thud.
$ git clone -b angstrom-v2018.12-thud git://github.com/Angstrom-distribution/meta-angstrom.git yocto/meta-angstrom4. Clone the Raspberry Pi Layer. This is a Machine(BSP) Layer.
$ git clone -b thud git://git.yoctoproject.org/meta-raspberrypi yocto/meta-raspberrypi5. Finally clone the Qt 5 Layer. This software Layer is indirectly needed.
$ git clone -b thud https://github.com/meta-qt5/meta-qt5.git yocto/meta-qt56. Open a terminal and initialize the build environment. This is needed whenever you open a new terminal to work with Yocto. In addition, it will perform a first time configuration and populate some files we will change next.
$ cd yocto
$ source oe-init-build-env7. Even though you've added 3 layers inside Yocto's folder, you need to tell it explicitly to use them. This is done in the bblayers.conf file. I'll also add a few other layers that come by default and will also be needed to build the system.
~/yocto/build/conf/bblayers.conf
# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"
BBPATH = "${TOPDIR}"
BBFILES ?= ""
BBLAYERS ?= " \
${TOPDIR}/../meta \
${TOPDIR}/../meta-angstrom \
${TOPDIR}/../meta-poky \
${TOPDIR}/../meta-yocto-bsp \
${TOPDIR}/../meta-raspberrypi \
${TOPDIR}/../meta-qt5 \
${TOPDIR}/../meta-openembedded/meta-multimedia \
${TOPDIR}/../meta-openembedded/meta-networking \
${TOPDIR}/../meta-openembedded/meta-oe \
${TOPDIR}/../meta-openembedded/meta-python \
${TOPDIR}/../meta-openembedded/meta-webserver \
"8. Now let set some global build configurations on local.conf.
~/yocto/build/conf/local.conf
[...]
#Change the target to the Raspberry
# MACHINE ??= "qemux86"
MACHINE ??= "raspberrypi3"
[...]
#Set the distro to angstrom
#DISTRO ?= "poky"
DISTRO ?= "angstrom"
#The single liner that tells yocto to use U-Boot in your build
RPI_USE_U_BOOT = "1"9. That's it. Let's build the system.
$ bitbake core-image-basebitbake is primary tool for building recipes in Yocto and core-image-base is the final recipe that describes a basic system. From it, all other thousands of recipes are chained and executed. You'll see a summary of the what will be done as bellow.
Build Configuration:
BB_VERSION = "1.40.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "ubuntu-18.10"
TARGET_SYS = "arm-angstrom-linux-gnueabi"
MACHINE = "raspberrypi3"
DISTRO = "angstrom"
DISTRO_VERSION = "v2018.12"
TUNE_FEATURES = "arm armv7a vfp thumb neon callconvention-hard"
TARGET_FPU = "hard"
meta = "my-layer:1cab405d88149fd6332546adb4a80ba68db3"
meta-angstrom = "angstrom-v2018.12-thud:873935c31bef51a4c2e1cb"
meta-poky
meta-yocto-bsp = "my-layer:1cab405d88149fd633267c6ab4a80ba68db3"
meta-raspberrypi= "thud:c71d79efc5a06a0c896c278c94f5b14413bb4d69"
meta-qt5 = "thud:81937713f774d7dea854b22987b3ef1ab1f1cdbe"
meta-multimedia
meta-networking
meta-oe
meta-python
meta-webserver = "thud:4cd3a39f22a2712bfa8fc657d09fe2c7765a4005"After the process is done. Check for the final image on following directories:
~/yocto/build/tmp-glibc/deploy/images/raspberrypi3/Angstrom-core-image-base-glibc-ipk-v2018.12-raspberrypi3.rootfs.rpi-sdimg~/yocto/deploy/images/raspberrypi3/Angstrom-core-image-base-glibc-ipk-v2018.12-raspberrypi3.rootfs.rpi-sdimgNote that the exact name of the image can differ depending on the version you're building. Look for the file ended in .rpi-sdimg.
10. Flash the image and see the results! Use whatever method you prefer to transfer the image to the SD Card.
I hope after reading this post you'll be encouraged to try to build your own systems taking advantage of Yocto. It's a good way to learn what are the building blocks that constitute a for a Linux system. In the next posts we will explore more advanced concepts like cross compiling your apps to the target, flashing custom Linux kernels and finally we'll begin to build our own customized Linux Distro.
Meanwhile, happy coding!