Kernel downstream merge/patch strategy of F&S

Basic release cycle

As our boards and modules are mainly based on System-on-Chips (SoCs) from NXP, we are also following the release strategy of NXP. That means most of our boards use the NXP-specific kernel. NXP always uses LTS kernels for their products, typically those that also have support for Android.

Why are we not using the mainline kernel? Well, when a hardware manufacturer like NXP develops a new SoC, they have to port and implement quite a lot of device drivers. This is a task of many months. If the kernel version would change during this time, this would be an additional difficulty. Therefore when they begin, NXP takes a then up-to-date mainline LTS kernel and then does all the porting on exactly this version until everything is finished. So after a year or so, NXP has a working kernel with all the support for the new SoC. This is the so-called manufacturer kernel, with still the same version as when NXP started. But in the meantime, the mainline kernel has released about six new versions. So at this point of time, the manufacturer kernel is at least six versions behind mainline, but it is the only kernel that has support for this new SoC. So when F&S designs new boards and modules with this new SoC, we depend on the manufacturer kernel, because we need its device driver support.

Then we port this code to our new boards based on this SoC, which again takes some time. Especially with the switch from 32-bit (i.MX6/7) to 64-bit CPUs (i.MX8), this was quite an effort, because the boot process got a lot more complex. There is not just U-Boot and Linux anymore. There is also the ARM Trusted Firmware (ATF), the Trusted Execution Environment (opTEE), the DDR4 DRAM has higher frequencies and requires board-specific timings now, there is a DRAM firmware, and some additional on-chip controllers like System Control Unit (SCU), security cores (SECO or Secure Enclave) and HDMI all need their own firmwares.

In the way how NXP has designed the boot process, the developer of the system software has to deal with all this low-level stuff, which would be you. And this is rather complex, because even within one board family, files like the DRAM timing may need different variants, depending on the RAM size and the RAM chip manufacturer. And you don't even have all the necessary information, because it is only relevant for us, the board manufacturer. For example you are simply not interested in what type of DDR RAM or eMMC Flash we use. But nonetheless you would have to deal with this. So NXP's way is not suited very well for cases, where the task is split in two parts, like us as board manufacturer and you as system integrator.

So we decided to do this slightly differently. We hide all the low-level stuff in a first-step bootloader called NBoot, that is already pre-installed on all our boards. This is only changed by us, you do not need to care about it. You only have to think about U-Boot and Linux. If you are familiar with our i.MX6 boards, you already know this, as it was the same there. So from the outside, the handling stays the same as on earlier boards, only internally, NBoot has completely changed. This makes it easier for you, and it allows us to continue providing architecture based releases, where only one version of NBoot/U-Boot is sufficient for all our boards based on the same SoC.

So after a while, we have our version of a kernel (and U-Boot), based on the NXP version. And of course these releases start with the newest boards. Then we do releases with this version for all our remaining platforms. And this is a lot of work: fsimx6 Buildroot and Yocto, fsimx6sx Buildroot and Yocto, fsimx6ul Buildroot and Yocto, fsimx8mm Buildroot and Yocto, fsimx8mn Yocto, fsimx8mp Yocto, fsimx8ulp Yocto and now also fsimx93 Yocto, just to name the most important ones.

In parallel, we start working on a new kernel and for new boards that are in the pipeline. This is typically the next NXP kenel, but if the previous development and all the releases took longer, we sometimes skip an NXP release. This explains the steps in the kernel number we have releases for. They are always the same as NXP's versions, but probably skipping an NXP version now and then.

Newest releases

Our current releases are based on kernel 5.15 on all our platforms. The neweset SoCs that we have are i.MX8ULP and i.MX93. Support for these chips was added after 5.15 in 6.1, so we made pre-releases based on the newer kernel 6.1. They are called pre-releases, because they are not yet merged with our regular software branch and they still use the original NXP way of booting. Our current work is to update our regular release version to the same version. However NXP has released 6.6 in the meantime, so we are skipping 6.1 as a regular release and are preparing all our next releases based on 6.6. Which is why we have a pre-relase for i.MX93 based on 6.6 already, but this is also still the NXP way of booting. The merge to NBoot is still in progress.

Unfortunately, the boot process for these new CPUs has changed again. That's why we had to improve our NBoot concept again to also support i.MX8ULP and i.MX93. This is finished now and we are preparing the regular releases for all our platforms based on kernel 6.6. We will start with i.MX93 and i.MX8ULP, because we only had pre-releases there and we want to have a regular release for them as soon as possible. Then i.MX8MP and all other architectures will follow. So for now, 5.15 is the newest version for i.MX8MP, but a 6.6 release is on its way and will be available by the end of the year.

Kernel patchlevel

NXP supports a release for a specific kernel version for a year or so, and during this time, they also update the patchlevel of the kernel version from time to time. But then after that, the patch level stays fix. This is why we decided, to update the kernel patchlevel on our own, independently of NXP. This is why you can find a newer patchlevel in our current releases.

Bringing patches upstream

We would actually like to bring back patches to the upstream projects. But there is an obstacle that we have not yet been able to overcome. NXPs releases are already some versions behind mainline, we are again some weeks or months behind NXP. But the mainline projects only accept patches relative to their newest versions. But we do not have these neweset versions running on our boards. So we actually never managed to prepare any patches that could be accepted by the upstream projects.

Fortunately, NXP is getting closer and closer to mainline. For example with U-Boot, they switch quickly to the same .04 version after mainline U-Boot is out. We also try to catch up. Our hope is to bring at least some of our changes to mainline U-Boot in the future, and maybe also to Linux some day.

Mainline kernel

Of course, after a SoC is in the market for some time, the SoC specific support gradually also migrates to the mainline kernel. Therefore, theoretically, we could also use the mainline kernel for our older boards. Up to now, we avoided this, to have exactly the same software version for all of our boards. But we are actually thinking about switching older architectures to mainline. This would allow users to update the kernel more independently from us. We will test this in the near future and maybe you'll also see mainline based releases for older boards, soon.

History file

Actually, the history file is obsolete now. In the past, our releases always updated all sub-packages, i.e. U-Boot, Linux and Yocto or Buildroot. So if we had a release for one architecture, and then a release for the next architecture, and there were no changes needed, we nonetheless had to update all the history files, resulting in new versions for all sub-packages.

Unfortunately, this concept made it rather difficult to have point releases as quickly as possible. A point release is a release, where only minimal changes are done, for example to fix a severe bug or add support for a new board revision. Here, changes are typically only in one sub-package, e.g. only in Linux. But we had to update *all* sub-packages, and it was not visible anymore, that for example U-Boot was the exact same version as before.

A year ago, starting with our 5.15 releases, we changed our release strategy considerably. We started providing our source code repositories on Github and combined the packages in a different way, pulling in the code from Github. From our internal point of view, all the sub-packages like U-Boot, Linux and Yocto/Buildroot are released on their own now, and a full release to the public, represented by a new releases-repository, only pulls in all the corresponding sub-packages. If we now do a point release that only differs in Linux, then we can pull in the exact same U-Boot as before, the exact same Yocto/Buildroot as before, and only need a newer Linux package. Unfortunately, this means we can not update the history file anymore, because the sub-packages do not know to which architecture release they currently belong or will belong in the future. In our example of the point release, U-Boot intentionally stays unchanged and therefore the history file, too. i.e. the history file cannot reflect the real history anymore.

We are still thinking of a good way of how to make the information of the history file available somewhere else. Part of it is in the Readme.txt of the release, part of it is the commit log of the git repositories. So we kept the history files for now, but they are obsolete and may disappear soon.

Your F&S Support Team