memory management problem after upgrade of BuildRoot V1.2 to BuildRoot V2.0

Thanks guys for update, have to admit that Imiss release of newer NBott, I use NBoot VN25 from BuildRoot V2.0 and V2.1. So I upgrade NBoot to VN27, cleared flash, upload UBoot and then upload via USB kernel and rootfs. But behavior seems to be the same :(.
Any other idea to check or try, Im out of ideas slowly.

Thanks anyway, appreciate any response.

Hello guys,

Sorry for delay of my response.

Now I have over 15 boards, tested on upgrade and same software, with same result. Last week I brought two devices with older version of SW, did upgrade and let them run. One with revision 1.31 seemed to be stable, second one with revision 1.3 worked only for few minutes before memory corruption messages and watchdog reboot.

Colleague in workshop tests another 15 boards, so Im not sure how many of them cause same malfunction.

At this moment four boards should be on way to you for test or checks.

After yesterdays result I had to stop upgrade preparation and found another way to solve this out. So if you have any advice what to try, test or do, any ideas are welcome.

Thanks and have a nice day.

Hi guys,
thanks for update. Im afraid applying patch to FS2.0 kernel does not solve problem. But it does make sense, so I’ll keep it in.

Patch testing configuration was NBootV27, UBootFS2.0 /UBootFS2.1 with kernel FS2.0 and original binary of kernel FS2.1. Module revision 1.3.

In the meantime we did some simple tests. Whatever cause these troubles, it seems to be NAND independable. Running rootfs from mmc card ( and kernel too last week, but now I wasn’t able to repeat this, huh ) did very same behavior.

Forbidding all our application we were able to confirm that memory corruptions can be invoked only thru running mplayer instance which plays live video stream from IP camera. Original buildroot’s MPlayer behaves same way as we use our compilation of MPlayer.

Simple streaming into file ( using ffmpeg ) does not cause any memory corruption or instabilities, so as playing any video file by MPlayers. Based on these I would dig into mplayer ( and Live555 which is used ), but on some modules takes hours to complete system crash, on some minutes ( as actual tested ) and on some there is absolutely no problem ( with same images ).

Looking for some any other stress tool ( or memory tester instead of one in NBoot ) I found
quoting:
All the really nasty errors are dynamical errors, which usually happen not with simple read or write cycles on the bus, but when performing back-to-back data transfers in burst mode. Such accesses usually happen only for certain DMA operations, or for heavy cache use (instruction fetching, cache flushing).

Generation of BUG kmemleak_object: Poison overwritten messages indicating corruption of slab pages like
Object 0xba8dfa80: 6b 6b 6b 6b 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkk..kkkkkkkkkk
during receiving and playing of video stream makes me believe that something can happened during system stress. So we will try to make some stress tests and let you know if we found anything suspicious.

If you have any advice where to look, what to check, other unreleased patches or we could send you something to help, let us know .

Thanks a lot anyway and have a nice day.

Yes, youre right.
MPlayer itself doesn’t use HW acceleration, its video canvas is rendered just by software. It’s not able to play any big resolution in extra quality, but its easy to maintain mplayer instance as process.

Board has to render four independent streams from IP cameras ( each with resolution around 400x200 ) and switch layout according input events. So it was easy maintain one instance of MPlayer for each visible camera.

I did experiments with gstreamer too, but found issue with four views added into one screen with blinking according to refresh rate only of the first one, so other views were snatchy. But its quite a time, too.

Right now, gstreamer doesn’t cause instability, I saw memory corruption only once, and stress tool does not invoke any bad allocation either.

OK, Ill try to find what can I do before total migration to newer version FS2.1.
But if you find anything – Ill be here .

Hi guys,

So after a while Im back. As to your recommendation, I did migration to new version FSV2.1.

After recompilations and testing I found few thinks.

On newer QBliss boards works very well. It runs smoother, boots faster and looks stable. Few devices worked thru two days without crushing. That’s on boards which came as board revision 1.31, equipped with imx6 revision 1.5.
On the other hand, older ones equipped with imx6 revision 1.2 seems to crush once upon while, with same software ( and proper upgrade to VN27 NBoot, UBoot2.1 ). Crush messages usually starts with

Unable to handle kernel NULL pointer dereference at virtual address 00000000 ( or some other very low address ),

but stack looks almost different. Sometimes one core just freeze, without any debug message.

I did some digging around, try to use some errata workaround available in kernel, but without reasonable improvement.Same behavior using defconfig versions of configuration.

All I saw led me to suspicion if there could be issue with mapping of physical pages to virtual memory ( conjunction with MMU ) or calculation of address space, or with floating point unit on early revisions of imx6. We used another toolchain with softfloat, if I remember well.

So, are you guy aware of some changes during revisions, which could affect that all works well on quite new revisions ( of board and imx6 ) and all of older ones crushes ? Or what to try, Im out of ideas what to reasonably try or where to look. By myself, I found some errata descriptions, but it doesn’t help me much.

We have to decide what to do during upcoming week, because situation looks unsolvable.

Thanks, appreciate any advice

Hi to all of you guys,

Thanks very much for response.

As we did some test ( mostly just functional test with our application ) we were able to check which board should probably work and which don’t.

Comparing board releases, i.mx chip series lead us to the fact that there is no direct relation between crashes and one “type” of chip. And so as to board revision and memory chip type. We can just notice older board ( originally with FS1.2 SW ) have higher probability of troubles ( after upgrade do FSV2.1 ).
We also found few new boards ( with FSV2.0 at origin ), according to serial and imx type numbers, which freeze applications or crash after few hours of testing. But older ones usually crashes after minutes.

Second remarkable thing is simple RAM test with build in memtester. Even its only able to test amount of free space in RAM intended to user usage, shows difference in modules behavior. Some modules shows ok status during all tests, cycles and reboots. These works in tests for 1 – 2 days without even sign of problem.
Other ones show usually many Failures in RAM values comparing. Every cycle or test on same device shows different offset address where failure was detected. These modules crashes system with memory allocations, NULL pointer issue, invalid instruction, segmentation fault and so on.

So if you say that there could be something with RAM powering issue, it does make sense to me. As I’ve been digging thru my tip was something with frequency or MMU, but I didn’t find any patch or other directly mentioning it.

I don’t believe that there could be so much board with corrupter memory chips and you are able to fix this with modified NBoot, its very good news. That’s the bright side.

I don’t want to sound ungrateful, but after next week we have to upgrade according to customer order all devices ( over four hundred devices ), means I have literary one week to prepare and fix all issues.

So, is there any chance to get some fix during next week ? It doesn’t have to be official version, even untested ( we can test on our boards ), I’m aware that its not easy to make this.

I take anything, anything, and if it’s true about powering, my little bit desperate situation could be expressed in classic quote

Help me Obi-Wan, you’re my only hope

Appreciate your effort, thanks so much, hope everything goes well.