Loggen met 8kHz op een ESP32-S3 in Arduino: het programma uitvoeren vanuit PSRAM (XiP)

The Jupiter's IMU is moving to 10MHz SPI on the next board so it can log at 8kHz, up from the 2kHz it manages on I2C today. Over on the Rocketry Forum someone fairly questioned whether an ESP32-S3 at 240MHz can really "log" at 8kHz, as opposed to just reading the sensor that fast. It's a good question and while I was confident we could do it, I hadn't yet verified it, so rather than guess we went and measured it properly. This is what we found, and at the bottom there's the rebuilt Arduino library we ended up with, if you want to try it in your own projects.

It isn't really the clock speed

The thing that stops most ESP32 projects logging quickly isn't the CPU, it's the flash. On a normal build the program runs straight out of the same flash chip you're logging to, through a small cache. Every time you write to that flash, the ESP switches the cache off on both cores until the write finishes, so any code that isn't already sitting in internal RAM simply stops. Not slows down, stops. A page write is around a millisecond, and erasing a 64KB block can freeze everything for 100 to 200ms. For a flight computer running a fixed rate loop, that's the real problem.

What we're logging

Each IMU sample is the gyro as three 16-bit integers and the acceleration as three 24-bit integers, the high and low g accelerometers fused into one reading, so 15 bytes a sample.

We started out planning to store the fused acceleration as floats, but integers turned out to be the better fit. The unit is exactly 1/2048 g, which is the low g accelerometer's own step on its 16g range (ST's 0.488mg is that number rounded), so its readings land on whole numbers and nothing the sensor measures is lost. Turning it back into g is a divide by 2048, which is exact, and 24 bits covers about ±4000g, far more than the 256g high g range needs. It's 3 bytes an axis rather than 4, and unlike a float it can be delta encoded later if we ever want to (more on that below).

Everything else the Jupiter logs (the 500Hz flight state, baro, magnetometer, GNSS, power and so on) adds about 20kB/s whatever the IMU rate, so the faster we go the smaller a share of the data it becomes: 40% of it at 2kHz, a quarter at 4kHz and only 14% at 8kHz.

IMU rate	IMU data (kB/s)	Other data (kB/s)	Total (kB/s)	With delta (kB/s)
2kHz	30	20	50	42
4kHz	60	20	80	64
8kHz	120	20	140	108

The other data figure is approximate. The with delta column is our estimate of the total to flash in the boost, where the motor's vibration makes delta encoding least effective, with only the IMU data delta encoded. On the pad and coasting it would be lower again. There's more on delta encoding further down.

How the Jupiter logs

• The IMU batches its samples in its own FIFO, so the loop isn't doing 8000 reads a second. It runs at 500Hz and pulls whatever is waiting in one burst each pass, which is 16 samples a pass at 8kHz.
• Every raw sample goes into ring buffers in the PSRAM. Nothing gets shuffled around, the rings are fixed blocks of memory with read and write positions, and the logger just walks along them in place. It also means the pre-launch data is already sitting there when launch is detected.
• A small 4KB buffer in internal RAM gathers the records and writes them to a dedicated raw flash partition. That partition is erased before the flight, so in flight it's only ever page writes, no erases and no filesystem in the way.
• After landing it compresses the whole flight into a single file, streamed through so it never needs the whole flight in memory, whatever rate it was logged at.

All of that still runs into the flash problem above though, which is where the next bit comes in.

Running the program from PSRAM

The ESP32-S3 can copy the program into PSRAM at boot and run it from there instead of the flash (Espressif call it XIP from PSRAM). Once it's doing that, a flash write no longer needs the cache switched off, so the loop carries on while the flash is written. The problem for Arduino users is that it's a setting not used in Espressif's precompiled libraries, so it's not something you can turn on from your sketch, so we rebuilt the S3 libraries for the Arduino core with it on.

ECC, because rockets get hot

While we were at it we turned on ECC for the PSRAM. Espressif rate the octal PSRAM in the S3R8 to 65C, and 85C with ECC on. 65C sounds a lot, but a dark rocket sat on a pad in the sun gets surprisingly warm inside, so for a flight computer it felt like the sensible place to start. It costs a sixteenth of the PSRAM, which we can easily spare.
It's worth noting this is the ambient temperature rating, not the chip's internal temperature rating. And as the Jupiter doesn't use RF (Wifi/BT etc) it will probably stretch a little further.

One more small one: where the write buffer lives

We found the 4KB write buffer matters too. If it's in PSRAM, the ESP-IDF flash driver copies it through a tiny 32 byte buffer and writes each 32 bytes as a separate command to the flash chip. If it's in internal RAM, it hands it straight over, 64 bytes a command, which is the most the S3's flash controller will take in one go. Each command costs about the same whatever it carries, so half the commands is nearly twice the speed: flat out we measured 209kB/s from PSRAM and 335kB/s from internal RAM. 4KB of internal RAM is nothing, so that one was an easy change.

The test

We added a test to the firmware that writes the flight buffer the same way the flight logger does while the 500Hz loop carries on running, then reports how badly the loop got held up. It writes at 55kB/s, then 166kB/s, then flat out, then erases everything it wrote. Those two rates were set from our earlier float plan (18 bytes a sample) for 2kHz and 8kHz logging, so they're a little above what the integer format needs, which makes the test a bit harder than real life. To be clear on one thing, the IMU on this board is still on I2C at 2kHz, so the faster phases are the flash load of faster logging, not faster sampling. Same board, same test, only the Arduino libraries swapped.

Each cell is the stock Arduino core first, then our rebuilt libraries with the program in PSRAM and ECC on. The loop should run at 500Hz, a pass every 2ms, and the worst gap is the longest time between two passes.

While the flash is	Loop rate	Worst gap	FIFO overruns
Writing at 55kB/s	349 → 499Hz	15.8 → 5.4ms	42 → 0
Writing at 166kB/s	193 → 499Hz	15.0 → 5.2ms	77 → 0
Writing flat out	196 → 499Hz	12.9 → 4.6ms	24 → 0
Erasing (64KB blocks)	17 → 499Hz	202 → 4.9ms	3 → 0

On the stock core the loop falls apart as soon as the flash is busy, it couldn't even keep up with the 166kB/s phase (it managed 160), and the erase stops it for 200ms. Every overrun is the IMU's FIFO filling up and samples being lost. With the program in PSRAM it holds 499Hz through all of it, including the erase. The worst gap of about 5ms is just the loop's own slowest pass (it does that with no flash writing going on at all), the IMU's FIFO soaks it up, and nothing is lost.

Delta encoding

Delta encoding means storing how much each reading changed since the last one rather than the reading itself. At these rates one sample is very close to the next, so most of the changes fit in one or two bytes rather than two or three. This is where the integers earn their keep: the change between two integers is itself a small integer, whereas a float's bits don't shrink when the value barely moves. In 1/2048 g steps, a change of up to about 4g between samples fits in 2 bytes.

How much it saves depends on what the rocket is doing. Sat on the pad or coasting, most samples would drop to 6 to 8 bytes, but in the boost with the motor shaking everything it's more like 11, and the boost is what matters as that's where the logging rate is at its highest. That's our rough estimate rather than a measurement for now. The Jupiter doesn't delta encode the flight buffer at the moment, as it doesn't need it for speed. Where it would help is recording time: the 7.8MB flight buffer holds about 58 seconds at a full 8kHz raw, or about 75 seconds delta encoded, before the hybrid logging steps the rate down.

So how fast could it log?

Putting the measured flash speeds together with the data sizes above gives the most each setup could keep up with:

Setup	Flash, flat out (kB/s)	Max rate, raw / delta (kHz)	8kHz data, raw / delta (kB/s and share of flash)
Stock core, buffer in PSRAM*	160	9 / 12	140 / 108 88% / 68%
Stock core, buffer in internal RAM*	306	19 / 25	140 / 108 46% / 35%
Rebuilt libraries, buffer in PSRAM	209	12 / 17	140 / 108 67% / 52%
Rebuilt libraries, buffer in internal RAM (the Jupiter)	335	21 / 28	140 / 108 42% / 32%

* The flash keeps up on the stock core but the loop doesn't. It dropped under 200Hz and lost IMU samples at every rate we tried, so in practice it can't log cleanly at any of these.

Raw is 15 bytes a sample, delta encoded is our 11 byte boost estimate, both include the 20kB/s of everything else, and they're the flat-out speeds measured with the loop running, rounded down with nothing kept spare. The 10MHz SPI bus isn't the limit at any of these either, it could carry around 59,000 samples a second.

So 8kHz needs about 140kB/s against the 335kB/s the flash takes flat out, about 42% of it without any delta encoding at all, or about 108kB/s (32%) with it. The CPU isn't working that hard either: at 2kHz most of each loop pass is spent waiting on the I2C bus, and on 10MHz SPI the same read at 8kHz should take around 0.3ms. The IMU itself tops out at 8kHz anyway, so the rest is headroom. It's still bench testing until the SPI board turns up, and the defaults will be lower anyway, with the logging dropping its rate after the boost, but the S3 looks well up to it.

Try it yourself

Seeing as we've done the hard work, the rebuilt ESP32-S3 libraries are in the downloads at the bottom of this post for anyone else using Arduino. They're for the esp32 core 3.3.12 from the Boards Manager only, and they replace this folder:

Windows  %LOCALAPPDATA%\Arduino15\packages\esp32\tools\esp32s3-libs\3.3.12
Mac      ~/Library/Arduino15/packages/esp32/tools/esp32s3-libs/3.3.12
Linux    ~/.arduino15/packages/esp32/tools/esp32s3-libs/3.3.12

On Windows, close the Arduino IDE, then open a Command Prompt (type cmd in the Start menu, not PowerShell) and paste this in. It finds the zip in your Downloads folder whatever the download added to its name, and it keeps your original folder as 3.3.12.stock:

cd /d "%LOCALAPPDATA%\Arduino15\packages\esp32\tools\esp32s3-libs"
ren 3.3.12 3.3.12.stock && for %f in ("%USERPROFILE%\Downloads\*esp32s3-libs-3.3.12-xip-ecc*.zip") do tar -xf "%f"
rmdir /s /q "%LOCALAPPDATA%\arduino\cores"

The last line clears the IDE's cached copy of the compiled core so it rebuilds against the new libraries. To go back to stock:

cd /d "%LOCALAPPDATA%\Arduino15\packages\esp32\tools\esp32s3-libs"
ren 3.3.12 3.3.12.xip-ecc && ren 3.3.12.stock 3.3.12
rmdir /s /q "%LOCALAPPDATA%\arduino\cores"

Don't use Explorer's Extract All on the zip, a few of the paths inside are too long for it and it will quietly skip files. The command above or 7-Zip are fine.

A few things to know:

• We've tested it on an S3 with 8MB of octal PSRAM (the S3R8), with PSRAM set to OPI PSRAM in the Tools menu. We haven't tried it on quad PSRAM, and as the program runs from PSRAM it won't suit boards without any.
• The program is copied into PSRAM at boot, so you lose that much PSRAM, plus a sixteenth of it for ECC.
• It's built with Espressif's esp32-arduino-lib-builder on ESP-IDF 5.5.5, the same as the official 3.3.12 libraries, with just two settings added. All the credit goes to Espressif. It's unofficial and as is, so keep your stock folder.

If you'd rather build it yourself, it's Espressif's esp32-arduino-lib-builder (Linux or WSL), checked out at the versions listed in the versions.txt file in the official 3.3.12 libraries folder, with these two lines added to configs/defconfig.esp32s3 before running the build:

CONFIG_SPIRAM_XIP_FROM_PSRAM=y
CONFIG_SPIRAM_ECC_ENABLE=y