TrueFuse 2E: O que Duzentos Voos Ensinaram ao Nosso Filtro de Fusão de Altímetro

The second part of the altitude fusion filter behind AltimeterCloud and the Nano. A slew witness for the notches 2C could not see, a burnout stamp that has to earn belief, a port to the device that had to be identical, and some of the many things we tried that did not survive the evidence.
It's been quite a journey with over 80 new bits and pieces tested. Visually checked against almost 700 flight logs and tested over 200 logs intensivly as we went sample by sample. Millions of logged samples checked and tested with every step of development. 

In August we published the story of TrueFuse 2C: why a barometer lies during a rocket flight, why we did not build a Kalman filter, and how a deterministic referee fusing the barometer and the accelerometer came to run identically on AltimeterCloud and on our altimeters. That article stands, and if you have not read it, the design arguments there are the foundation for everything here. This one is about what happened next: two hundred and one real flights, run through every candidate change and compared exactly, and the filter that came out the other side, TrueFuse 2E, now shipping in Nano firmware 1.63 and on the website.

And to the eagle eyed, yes 2D was a lot of work that we simply spent too long suffering with. As a result it felt much happier to ditch it and skip to 2E!

What the fusion is for, in a paragraph

A pressure sensor is the best altitude sensor on the board almost all of the time, and badly wrong in a few specific regimes: while the rocket is fast, when an ejection charge fires inside the airframe, and in a handful of stranger cases we listed last time. The accelerometer knows nothing about air pressure and everything about thrust, so a filter that carries the rocket's orientation, projects thrust onto the vertical and integrates it has a second witness for precisely the moments the barometer cannot be trusted. TrueFuse hands the estimate between the two on a schedule set by speed, because the port's error scales with dynamic pressure, guards the coast and descent with a trend sensor that freezes a fitted arc across an artifact, and declines cleanly when the evidence is untrustworthy. It runs live on the Mercury and Jupiter, and at save time on the Nano. TruePath, the post pass with the whole flight in view, runs after it on every device.

This chart shows the acceleration noise during ignition where the altimeter was loose and slipped to a new position.
The red dashed line shows the raw pressure based altitude that is effected by the motor thrust.
The green line shows truefuse 2E's corrected output to what the physics say was possible and real. 

This chart shows the ejection charge going off just before apogee. The red line shows the corrupted pressure based altitude.
The green line shows the TrueFuse 2E fixed path the rocket took.

The important part of this is that it is done live and without delay, it does not rely on post processing of the data. This allows flight computers to make decisions during pressure events that would normally not be possible.

 

How a change gets in

The rule that made 2C provable is the rule that made 2E possible. Every proposed change is made as one bounded mechanism behind a switch, run across a battery of real fleet flights, and compared to the previous revision sample by sample. Not statistically: byte for byte. A change that alters a flight it was not meant to touch is visible immediately, and a change that fixes the flight in front of us at the cost of three others is not a fix. Alongside that, every revision passes a causality check: the battery is re-run with the log cut off at four points and the output up to each cut must be identical to the full run, to the last digit, because a filter that runs live on a device cannot be allowed to peek at the future on a website where the future happens to be available.

The flight log battery grew from 196 flights to 201 during this work, and the additions were all flights we had looked at and said "that is not right": a Mercury with a corrupt IMU burst at launch, two Nanos with a flat spot in the climb, a flight whose device stamped launch a second and a half late. That is how the battery is meant to grow. Nothing is tuned to a flight; every flight that exposes a mechanism joins the jury.

This method has one property worth stating plainly, because it shaped 2E more than any single mechanism did: withdrawing a change is a normal outcome, not a failure. Several of the ideas below were built, run, measured and taken out again, and the code carries the number that killed each one.

The notch the hold detector cannot see

2C's trend sensor opens a hold when the barometer leaves its fitted path by more than six metres, on both the pressure prediction and the accelerometer's, and that catches the ejection artifacts that matter most, the tens and hundreds of metres. What it could not resolve were the small ones. A Mercury flight in the battery, 2836, shows a notch four metres deep at ten metres per second, three quarters of a second before apogee: too small to trip the hold, entirely capable of pulling the trace down and up again, and sitting in the hand-over between the accelerometer and the barometer, where the trend sensor's holds could not own the output anyway.

The mechanism that resolves it is a slew witness, and it is the purest piece of physics in the filter. Over any 40 ms, the barometer may not honestly fall further than the vehicle could have moved in that time, plus the barometer's own noise as measured in the air over the previous second. The vehicle's speed comes from the filter's own velocity witnesses; the allowance is speed times time plus half the acceleration times time squared; the noise floor is six times the median absolute step of the raw samples. When the newest raw sample is further below the sample 40 ms ago than that allowance, the fall did not happen, and a bridge opens on the trend the barometer was following before it. The bridge is a physics line: seeded at the trend's value, carried on the accelerometer's projection, capped at the height the trend's velocity could reach against gravity, and released when the raw comes back to it or when apogee is confirmed. It works during the hand-over as well as after it, because the fuse grants it the same authority it would grant a hold.

Its counterpart is a rule about precedence that we got wrong first. The slew witness fires early, three milliseconds ahead of the hold detector on two flights in the battery, and our first version let the bridge keep the output when both saw the same event. The battery said otherwise: on those flights the bridge's ceiling, computed from a trend slope that under-reads near apogee, stopped the line 1.4 metres short of where the vehicle went, and the hold, whose arc has no ceiling and releases when the pressure crosses it, had been doing the job better all along. So a hold now outranks a bridge. The bridge exists for the notches the hold detector cannot see; where it can, its hold owns the output.

As always the fact that this is built to run live without knowing the future made it all the more fun. What's easy to see by eye seeing ahead on the chart, is a whole different story once you are asked to point to the end of a dip with your eyes closed. 

A third piece went the other way. 2C's trend sensor had a dip test, opening a bridge when raw fell two metres below the trend. Its only opens on the whole battery were 2836's notch, which the slew witness now sees 110 ms sooner, and one false open 270 ms after a flight's true apogee, where a two second quadratic still read fifteen metres per second of climb and the honest descent looked like a dip. It printed a 0.75 m hat on a flight 2C leaves alone. It has stood down; the code is still there, switched off, with the number beside it.

A burnout stamp has to earn belief

The filter's schedules turn on the burnout stamp, and the device's live detector is usually right. When it is impossible, zero, after apogee, or after the confirmation, a repair law rebuilds it from the accelerometer: ignition is the first two consecutive samples over 1.5 G, burnout the first three under 1.0 G after it. 2C believed whatever that law found. Flight 1552, a Mercury, showed why it should not: a corrupt IMU burst at launch detect, 6.1 G written twice with a 1700 degree per second gyro, was accepted as a motor, burnout was stamped at 160 ms in the middle of the real burn, and the fuse handed over to the barometer inside the burnout dip, calling the apogee 54 metres on a flight that went to 100.

2E's repair requires a candidate burn to have produced velocity, by the fuse's own false-trigger bar, before it is believed, and refuses any candidate that ends after apogee. That second guard has already earned its place on hardware. On a Nano hand throw during testing, the catch at the end put a 24 G spike into the accelerometer, the live detector took it for the end of a burn and stamped burnout at 1,726 ms on a flight whose apogee was at 1,139, and the repair rebuilt the stamp from the throw itself: 486 ms, where the hand let go. The website, given the same log, arrived at the same millisecond.

Two smaller pieces of launch discipline came from the same battery. The pad stage, which holds the output still while the rocket sits on the rail, used to end only at launch detect; on a flight whose device stamped launch 1.5 seconds late it was still live 25 metres up, and when the burn tailed briefly back into the still band it pinned the output to the pad for three samples, a 33 metre blip to zero. It now ends the moment the airframe has been out of the still band for longer than any hold may last, whatever the stamp says. And a fuse that fires on the pad and decides it was a false start now forgets everything it learnt, because on four flights it had left the hand-over band anchored at a standstill and carried the accelerometer 25 metres over the barometer to the close.

The hand-over, smoothed twice

Two Nano flights showed a flat spot in the climb: twenty or forty samples held level a moment after the fuse settled. The cause was the merge band, the speed range over which the accelerometer's weight fades to the barometer's, being anchored at a fixed time after burnout. A fuse that settles before that time re-based its band mid-blend, the weight fell from 0.36 to 0.003 in one sample, the target stepped 2.3 metres, and the climb guard caught the step and held it. The band is now anchored at the first sample of the blend, and the flat spots are gone: 21 held samples to 0 on one flight, 41 to 0 on the other, apogees unchanged.

The last of the blend had the same character in miniature: a linear fade that clamped at full weight with its slope intact, so the trace arrived on the barometer at an angle. It now fades on the same eased curve as the rest of the weight, front-loaded and tangent at the join. A smoothstep was tried first and kept the angle; an additive closing law, the output as pressure plus a decaying disagreement, was tried after and withdrawn when it put 25 metres on one flight and 21 on another. Both are recorded in the file with their numbers.

And one change that no flight asked for but the sample rates did: the pressure velocity the fuse watches was measured over a fixed count of samples, which is a different span of time at 32 Hz and at 400 Hz. It is measured over time spans now, so its lag means the same thing on every device.

Holds that let go

Three changes to the trend sensor's holds came from watching how they ended. A hold used to release only when the raw had travelled with the prediction for a while, in slope as well as value, which is the right test for an artifact's tail and the wrong one for the common case where the pressure simply crosses the held arc and carries on. A hold now releases once the pressure has been across its arc for 40 ms, and merges over 100 ms instead of 200. Descent holds land under two metres from the barometer where they used to land nearly five. Before apogee is confirmed, the arc a hold freezes onto now ignores its newest 300 ms, the rule the descent already had, so an ejection spike at apogee cannot tilt the arc upward; pre-apogee holds release within 0.8 metres of the barometer instead of 1.7. And the climb guard, which stops the output falling while the accelerometer says the rocket is climbing, no longer counts samples the launch close is still walking out as evidence of climb; on one flight that had built a 73 metre diagonal off a 25 metre peak.

Data that is not data, again

Mercury firmware widens the IMU's ranges at launch detect, and the first samples after the switch carry the old range's reading scaled by the new one: eight times. A rail of 4 G reads 31.7 G, a rail of 250 degrees per second reads 2,000, and the row is often written twice. Sixty-one of the ninety Mercury flights in the battery have it, always inside the first 60 ms. The gyro half of this was the "ignition slip" 2C already handled; the accelerometer half was being integrated as thrust, and on one flight a single row of 33.7 G was worth 12 metres per second to the fuse, fifteen metres over the barometer for the whole climb.

A filter-level rule was tried first and failed, for a reason that is worth keeping: the website re-times the IMU columns onto the pressure timeline before the filter runs, and that re-timing had already interpolated the corrupt rows into their neighbours by the time the filter looked. The repair had to happen in the feed, ahead of the re-timing. It does now, on the website; the firmware fix is to discard a few samples after a range switch, and that is on the Mercury's list.

What was tried and left out

The 2C article told the story of the slipping altimeter, the loose unit shoved into a new seat by the thrust load. 2E spent a fair amount of its development on two ideas for doing better than tolerating it, and both are worth describing because of how they died.

The first was an up-correction of the carried vertical: when the slip watch fires, rotate the vertical by the measured slip so the projection is not charged the phantom tilt. It fixed the flights it was built for and printed a fifteen metre hump on another, because a rocket that is slipping and a rocket that is genuinely tipping still look alike from inside, and the battery does not accept a fix that breaks a flight it was not for. The second was an impulse repayment: measure the velocity the fuse lost during the slip's magnitude bridge and pay it back afterwards. Built, run on all twenty-five slip flights in the battery, and the repayment came out at 0.00 metres per second on every one of them, because the magnitude bridge was already carrying the velocity the correction wanted to restore. Nothing to repay. Both mechanisms are out of the shipped filter, and ignition slip is handled as 2C handled it, by the magnitude bridge and the thrust-vector re-anchor.

The list of things tried and withdrawn during 2E is longer than the list of things kept: the slip up-correction, the impulse repayment, the dip test, the smoothstep fade, the additive closing law, a filter-level rule for the range-switch rows, a version of the slew witness that opened a hold rather than a bridge (it self-cancelled on the very notch it was for and fired on descent wobbles), and a version of the bridge that seeded its velocity from the era's own witness (every velocity witness near apogee under-reads, and a lower seed caps the line short). Each is a sentence in the source with the flight and the number that ended it. That is the referee's discipline applied to the referee.

TruePath, and finishing the bridge's job

TruePath moved five revisions while this went on. It now recognises a gap in the log and passes the affected samples through rather than smearing across the hole, judged against the sampling rate around the gap so that hybrid mode's slower logging is not mistaken for one. A flagged run that reaches the start or end of a record is left alone rather than held flat, after one flight printed 132 metres while the barometer had fallen to 26. And it learnt the one thing it needed to from 2E: the live filter cannot bridge a notch before it knows the notch is there, so the first samples of every notch, typically 20 ms of them, are on the trace 2E hands over. TruePath receives the bridge events, the moment the notch began, the moment the bridge opened and the trend it rode, and sets those samples onto the trend and the bridge's own line. Nothing of the artifact survives into the saved file. You can see it in the first figure: the dashed line is the finished trace.

Time itself, again

Last time the discovery was that two sensor pipelines have two clocks. This time it was smaller and stranger. The filter's sample times are the log's seconds multiplied by a thousand, in double precision, and about one time in a thousand that product lands a hair below the integer: 1.871 times 1000 is 1870.9999999999998. Every test of the form "is the time at or past the burnout stamp plus 500 ms" then fires one sample late on those times. It is invisible in a chart and harmless to any flight, but it is not reproducible by a device that holds integer milliseconds, and the whole point of the port you are about to read about was that the device and the website agree exactly. So the website now rounds its sample times to the microsecond, on both filters. Eighty-nine battery flights moved by a sample somewhere, none of them by more than a centimetre except one broken flight inside a pressure spike, and no apogee moved.

A second timing fact came to light from the same comparison. Since firmware 1.62 the Nano has fed its filter apogee stamps from a median probe over the recorded trace, while the website was still reconstructing them by re-running the older firmware's live detector. Both were reasonable; they were different, by up to 400 ms on the confirmation. The website now takes the device's own stamps from the log header where a log carries them, so it runs the filter on the inputs the device ran it on rather than a reconstruction of them.

Getting it onto the Nano, and testing it blind

The porting rule from last time was absolute and it did not get any looser: the device runs the same law, not a resemblance of it. This time the port had a head start. 2C's C++ on the Nano was a statement-for-statement transcription of the reference, so 2E's port was the difference between the two reference filters, thirty-three hunks of Perl (I know... but in my defense Perl is my native programming tongue so it still pops up here and there!), applied to that C++ in the same discipline. Before a line was written, the reference itself was frozen: every mechanism that had been switched off was removed, every switch inlined, and the frozen file proven byte-identical to the full one on all 201 flights. That file was the specification. Nothing that was not in it went anywhere near the firmware.

Then the blind test. The website's harness was made to dump, for every battery flight, exactly the arrays and stamps it hands its filter. The C++ was built on a desktop, run on those files, and its output compared to the reference sample by sample without either side seeing the other. The double precision build is identical to the reference to four decimal places on all 201 flights, for the fused trace and for the final trace after TruePath. The float build, which is what the device runs because the Nano's ESP32-S2 has no floating point hardware at all, holds every apogee within a millimetre and the trace within two centimetres at its worst sample.

Getting the float build there taught us three things we did not know. Double precision was tried for the filter's state and then for its fits and bought nothing measurable, while growing the firmware by 33 KB and overrunning the flash partition; what the fits needed was conditioning, centring the altitude column before forming the normal equations so single precision was not subtracting near-equal products of hundreds of metres, and that mattered more than double did. The sample times had to be exact milliseconds in every build, because a float seconds value times a thousand is not, and that moved the bridge's trend fit by a sample and its slope by 0.7 percent until it was fixed. And the third we would rather have known in August: the 2C pass never freed its ring buffers. On every save it left about 45 KB behind, which on the S2 sat harmlessly in internal RAM until 2E's larger set spilled into PSRAM, where the leaked blocks landed in the middle of the space the log writer needs in one piece, and a 35,000 sample flight failed to save. Found with a counting allocator on the desktop, fixed by freeing the rings with everything else, and the device now uses about 230 KB less at its peak than 1.62 did at the maximum log length.

Eight throws on two devices, at every log length up to the 38,000 sample maximum, all sit on the website's line like that. The catch tried to pass itself off as a motor twice and was turned away twice. A 2E mechanism the throws were never designed to exercise, the ignition slip bridge, fired on one of them when the gyro spiked at release, on the device and on the website identically. What the throws cannot do is fly, and the bridge and the holds have not yet run on the device with a real ejection charge behind them. They have run, in code identical to four decimal places, on the two hundred flights that had one. The first real flights on 1.63 are being watched for exactly that, and the raw column in every log is untouched by any of this, so the evidence to judge the answer ships inside every flight, as before.

What is next

The same logic as last time: better evidence first, cleverer maths second. Sample times to a tenth of a millisecond on both sides, because a 2.5 ms cadence written as 2, 3, 2, 3 carries twenty percent of artificial jitter into every integration. A quicker barometer: the sensor's internal filter at coefficient 3 rather than 7 halves the pressure lag for half a millimetre of noise, and on the bench everything downstream preferred it, but it goes in after a flight campaign rather than before one. TruePath's smoother running on the same time scale on the website as on the device. A second gyroscope range, a magnetometer where the board has one (The Jupiter does), and the rigid-mount detection we described last time, all still on the list. And, before any of it, one real flight through the bridge.

TrueFuse & TruePath have been turned into library files so they can be included with ease in to any firmware.

TrueFuse 2E revision 37 and TruePath revision 6 are what the website runs (at the time this was published) and what Nano firmware 1.63 carries. The 2C article, with the design arguments this one builds on, is here.