Configuring ATF for power optimization

    I am trying to reduce power consumption of the PicoCoreMX8MP while the Linux kernel is in suspend mode and the Cortex-M7 core is in RUN mode.


    Therefore, I'm trying to follow the suggestions in AN13400, which require changes to the ATF.


    If I understand correctly: to integrate these changes, I must rebuild Nboot with the adjusted ATF. My question is then what is the recommended way of doing this?


    Do I simply compile bl31-imx8mp.bin from the adjusted ATF along with the other files mentioned in u-boot-fus/board/F+S/fsimx8mp/nboot/Makefile (shown below) and then run the nboot make target from the root of u-boot-fus? Is there a yocto recipe or other reference for this?

    Code
    1. NXP_DDR3L = ddr3_imem_1d.bin ddr3_dmem_1d.bin
    2. NXP_LPDDR4 = lpddr4_pmu_train_1d_imem$(DDR_FW_VERSION).bin lpddr4_pmu_train_1d_dmem$(DDR_FW_VERSION).bin \
    3. lpddr4_pmu_train_2d_imem$(DDR_FW_VERSION).bin lpddr4_pmu_train_2d_dmem$(DDR_FW_VERSION).bin
    4. NXP_DDR4 = ddr4_imem_1d$(NANO).bin ddr4_dmem_1d$(NANO).bin \
    5. ddr4_imem_2d$(NANO).bin ddr4_dmem_2d$(NANO).bin
    6. NXP_ATF = bl31-imx8mp.bin
    7. NXP_TEE = tee-fsimx8mp.bin

    The idea of NBoot is to hide all the low-level stuff from users so that they can concentrate on U-Boot and Linux and do not have to bother about DRAM settings, board configurations and firmwares. So NBoot is basically meant to be compiled by F&S, not by customers. But I can understand the reason for your need to do this. Maybe you can do your modifications in a rather generic way. Then when you are done, and if it's OK for you, we can add your improvements to our regular NBoot. This would save you some work in the future (see below).


    To compile NBoot, you just have to rename bl31.bin from the ATF build to bl31-imx8mp.bin and store it to board/F+S/NXP-Firmware. Please note that we have a slightly modified version of ATF, so start with our github version, not the original one from NXP.


    Then copy the DRAM-FW/Training data to this directory, too. They are part of the firmware-imx package. from NXP.


    For a regular NBoot, you won't need TEE. A Trusted Execution Environment without Secure Boot does not make much sense.


    Of course U-Boot must be configured for fsimx8mp, i.e. make fsimx8mp_defconfig. Then simply call make nboot. This will compile most of U-Boot, then SPL, then all the other sub-images (configurations, DRAM timings, etc:) and finally assembles the NBoot image. Please change NBOOT_VERSION in board/F+S/fsimx8mp/nboot/Makefile to make it different to our releases, e.g. add a prefix or suffix or similar.


    Then install NBoot with fsimage save in U-Boot. Also be aware that if your modifications cause a failure of ATF, so that U-Boot is not started, you have to re-install the software using the UUU tool or our Recovery Tool.


    We are currently in the process of writing a documentation for NBoot that will show all the internal details. But this takes a few more days until it is finished. It will only be availbale on request, so leave me a message if you want it.


    Please note that using your own NBoot has some severe disadvantages. NBoot always handles DRAM and flash initialization. So for example, if a DRAM chip goes end-of-life and we replace it with a different one, we will most probably need new DRAM timings and therefore a new board configuration to support this. So all new boards that are shipped from then on, have the new NBoot preinstalled. But if you replace this version with your own old version, then booting will most probably fail, because your version does not know about the new DRAM timings and board-config yet. In other words, for every change that we do in NBoot, you also have to rebuild your own version of NBoot. This will result in quite a lot more work that you have to do. If you also change stuff in NBoot other than ATF, things may get even more complicated.


    Your F&S Support Team

    F&S Elektronik Systeme GmbH
    As this is an international forum, please try to post in English.
    Da dies ein internationales Forum ist, bitten wir darum, Beiträge möglichst in Englisch zu verfassen.

    I can see that an LPA_ENABLE compile-time flag has been added since at least version lf-5.15.71-2.2.0 of imx-atf (which appears to be merged into atf-fus as well). E.g. in plat/imx/imx8m/ddr/dram_retention.c:


    Code
    1. #if defined(LPA_ENABLE)
    2. if (imx_is_m4_enabled() && imx_m4_lpa_active()) {
    3. /* disable the DRAM PLL */
    4. mmio_clrbits_32(0x30360050, (0x1 << 9));
    5. }
    6. (...)


    The changes as a result of having this flag defined seem to align with the changes suggested in AN13400. If you have the setup prepared already, would it be possible for you to compile a version of Nboot for PicoCoreMX8MP where the ATF is compiled with this flag defined?


    I certainly wouldn't mind if you are willing to share the WIP NBoot documentation you mentioned. It can be sent to nicv@itu.dk

    cortex-M4_Cortex-A53_low_power_use_cases-picocoremx8mp.zip


    Here's an update on this matter for any future readers. We have managed to get a working version which achieves sub-1W power consumption while keeping M7 and SYSPLL1 running for peripherals that need it. The attached file above has M7-side demo code attached. Here is an outline of the steps I've used. Take specifics with a grain of salt as they are written by a newcomer to the field who really has no business doing embedded!


    # ATF changes
    Compile with LPA_ENABLE and following patches:


    - Puts A53 in WFI so it can be woken up by M7

    Diff
    1. --- a/plat/imx/imx8m/gpc_common.c
    2. +++ b/plat/imx/imx8m/gpc_common.c
    3. @@ -268,6 +268,8 @@ void imx_set_sys_lpm(unsigned int last_core, bool retention)
    4. /* mask M4 DSM trigger if M4 is NOT enabled */
    5. if (!imx_is_m4_enabled())
    6. mmio_setbits_32(IMX_GPC_BASE + LPCR_M4, BIT(31));
    7. + else
    8. + mmio_setbits_32(IMX_GPC_BASE + SLPCR, SLPCR_A53_FASTWUP_STOP_MODE);

    - Prevent SYSTEM PLL1 from being bypassed and disabled so it can be used by M7 for peripherals that require it (e.g CAN with default timing at baud 1000000)

    Diff
    1. --- a/atf-fus/plat/imx/imx8m/imx8mp/imx8mp_lpa_psci.c
    2. +++ b/atf-fus/plat/imx/imx8m/imx8mp/imx8mp_lpa_psci.c
    3. @@ -355,11 +326,6 @@ static void bus_freq_dvfs(bool low_bus)
    4. mmio_clrbits_32(0x30360114, (0x1 << 9));
    5. NOTICE("disable sys pll 3 done \n");
    6. - /* disable the SYSTEM PLL1, bypass first, then disable */
    7. - NOTICE("bypass syspll1 \n");
    8. - mmio_setbits_32(0x30360094, (0x1 << 4));
    9. - NOTICE("disable syspll1 \n");
    10. - mmio_clrbits_32(0x30360094, (0x1 << 9));

    # Device tree changes
    Compile with SUPPORT_M7 and the following changes to be able to wake up A53 from M7 via MU:


    # IMX Mailbox driver changes
    Needs following patch to work with Linux 5.15 (enable and clear interrupt flag) (see `0002-imx8mm-gir-wakeup.patch` in https://community.nxp.com/t5/i…45/highlight/true#M193695 - this patch also works for i.MX8MP even though the author originally intended it for i.MX8MM):



    # M7 side
    The following must be set: SRC->GPR10 = 0x5555U (SRC->GPR10 is the LPA flags register)

    Also this for wake-up IOMUXC_GPR->GPR22 &= ~IOMUXC_GPR_GPR22_GPR_M7_CPUWAIT(0b0); (but that bit should be low by default)
    - This for the imx_is_m4_enabled function in ATF

    - SYSPLL1 is active, but CCM bypassed and disabled (so clocks using system PLL1 as clock root will not work properly) - E.g. when default UART1 clock is configured like this:

    C
    1. CLOCK_GetPllFreq(kCLOCK_SystemPll1Ctrl) / (CLOCK_GetRootPreDivider(kCLOCK_RootUart1)) / \
    2. (CLOCK_GetRootPostDivider(kCLOCK_RootUart1)) / 10

    - The following should be run after A53 gets put in WFI (prevent CCM bypassed and disabled in ATF may make this unnecessary, but we did not investigate further):

    C
    1. CLOCK_SetRootMux(kCLOCK_RootUart1, kCLOCK_UartRootmuxSysPll1Div10);
    2. CLOCK_SetRootDivider(kCLOCK_RootUart1, 1U, 1U);

    One can further modify the ATF or IMX mailbox driver to notify the M7 of the A53's state / provide a heartbeat. Alternatively, SLPCR (or other GPC registers) can be used to determine the state of the system.
    E.g. read (GPC->SLPCR & GPC_SLPCR_EN_DSM_MASK) >> GPC_SLPCR_EN_DSM_SHIFT


    ## How to wake up A53
    First the MU must be initialized: MU_Init(MUB). Then to wake up A53, either of the following can be used:
    - MU_TriggerInterrupts(MUB, kMU_GenInt0InterruptTrigger)
    - MU_SendMsg(MUB, 1, 2)


    # A53 side
    Currently only appears to work when echo N > /sys/module/printk/parameters/console_suspend is run before echo mem > /sys/power/state (debug is also enabled in the ATF)


    # Power measurements
    A few power measurements have been made. The measurements are taken with an Otii Ace Pro and the Otii 3.5.0 software. Furthermore, the measurements measure the power consumption at the supply input of a PicoCoreBBDSI Rev1.40 baseboard with debug serial connections active. Three scenarios are covered:
    - This example where M7 uses 24MHz oscillator (LPA-nboot-24mhz)
    - This example where M7 core uses SYS_PLL1 as root (as well as for UART1) (LPA-nboot-syspll1)
    - A default setup without the ATF changes in this example and the M7-core is not running (default-nboot-no-m7)


    In the low power state, the three configurations draw the following power:

    - LPA-nboot-24mhz: ~0.638 W
    - LPA-nboot-syspll1: ~0.822 W
    - default-nboot-no-m7: ~0.492 W


    Each example idles around 2W when A53 is in RUN mode.