We Said We Hit L5. Then We Tested With a Real Sensor Model.

A few weeks ago we wrote that we'd hit L5 autonomy — zero human interventions across a 16-scenario adversarial benchmark, a heterogeneous fleet of quadcopters and ground rovers. That result was real, and it's still in the repo. But it had an asterisk we didn't fully appreciate at the time, and chasing that asterisk turned into the most honest piece of engineering we've done this year. This is that story — including the parts that didn't work.

The asterisk

Our L5 controller is a reactive potential field: each agent reads its local surroundings — obstacle proximity, teammate positions, goal direction — and produces a velocity. To decide how hard to avoid an obstacle, it uses a number called min clearance: the distance to the nearest obstacle surface.

In the simulator, that number was the true perpendicular distance to the nearest obstacle surface — computed from ground-truth geometry. That is a quantity no real sensor produces. A lidar gives you a ring of range readings. A depth camera gives you a fan of them. Neither hands you “the exact distance to the nearest surface.” We were grading our autonomy on information it would never have on real hardware.

So we built a second sensing mode into the benchmark — --sensing realistic — that gives the controller only what a real sensor gives: the nearest return of the scan, no oracle. Then we re-ran the suite.

L5 became L3

Under realistic sensing, the fleet dropped from L5 to L3: two collisions, 81% mission success. The “L5” had been partly an artifact of idealized perception. That was a bad afternoon — but it's exactly the kind of thing you want to find in sim, on your own terms, rather than in a field test.

The first thing we did was make the benchmark honest permanently: realistic sensing plus per-agent, per-cause failure attribution, so every collision and every timeout names the agent and the reason. No more grading against an oracle.

The second thing we did was ship the exact validated controller onto the device code path — the same reactive controller and rule-based fleet advisor, running on the Jetson's flight software, proven byte-for-byte identical to the sim by a parity test. Whatever we validated is literally what flies. No “the real code is different” gap.

Getting back to L5 (deterministic)

With attribution, the failures were specific. Two mechanisms:

• Thin geometry margins. Some obstacles were placed with just enough clearance to work under perfect sensing. With realistic sensing the agent's trajectory drifts ~0.2m, and that ate the margin. The fix was the same minimal-perturbation geometry repair we'd used to reach L5 originally — now with a sensing-drift budget. Move two obstacle sets a little; collisions go to zero.
• Timidity near the goal. A few agents stalled ~2.5m short of goal, pinned behind patrolling intruders, just outside the range where the advisor's “near-goal lock” punches through. Raising that engagement range from 2.0m to 3.0m cleared all of them — without reintroducing a single collision.

Result: L5 under realistic deterministic sensing, both the oracle and the realistic modes, zero collisions, 100% success. Regression-locked with a test.

The nine things that didn't work

One deterministic run isn't “probably.” So we added a sensor-noise model — ±5cm Gaussian range noise plus 3% beam dropout — and ran a Monte Carlo: dozens of random seeds across all 16 scenarios. Deterministic L5 dropped to ~92% collision-free per run. The remaining collisions concentrated in two scenarios.

We assumed the culprit was fast dynamic intruders, and we spent real effort on it. In order, here is what we tried and what happened:

1. Surface reconstruction (fit the obstacle edge from adjacent beams) — no change.
2. Wider avoidance radius — no change.
3. Hold-last temporal filter to bridge dropouts — worse (ghosting).
4. Temporal median filter — much worse (lag: it trails the shrinking clearance during approach).
5. A scalar “something's close” gate on the near-goal lock — traded collisions for timeouts.
6. A measurement-uncertainty clearance margin — within noise, no real help.
7. A lag-compensated clearance estimator — looked like a breakthrough until a fair test showed the prototype had a state-leak bug flattering the numbers; done correctly, no help.
8. A retrained learned policy — a from-scratch RL run never converged; the existing noise-trained policies were dramatically worse than the rule-based controller.
9. A velocity-obstacle / ORCA solver — the correct tool for reciprocal agents, but it made things worse: the intruders are scripted and non-cooperative, so evading them in a tight corridor is worse than committing.

Nine approaches. None closed the gap. That's not a fun paragraph to write, but it's the true one — and it's what pointed at the real problem.

The diagnosis we'd gotten wrong

Every one of those approaches assumed a dynamic-obstacle problem. The attribution said otherwise: the collisions were labeled rover vs obstacle, not vs intruder. In the corridor scenario, the intruders fly at a different altitude than the ground rover — they can't even hit it. The rover was clipping the static wall under range noise. It was a geometry-margin problem the whole time, wearing a dynamic-obstacle costume.

The fix was the method that had worked twice already: widen the tight rover passages with a noise-drift budget (the quads fly over the tops, so it's rover-only and safe). Two scenario edits.

Result: 99.5% collision-free across 800 randomized noisy runs, up from 92.3% — deterministic L5 preserved, no controller changes. The remaining ~0.5% is one scenario (a rover that goes blind under GPS loss and wind simultaneously) where the obstacles are load-bearing for navigation and widening them backfires. We left it honest rather than force it.

Through the real autopilot

The last thing between an algorithm and an aircraft is the autopilot: actuator lag, control-loop latency, imperfect velocity tracking. So we ran the L5 controller through ArduPilot SITL — the real flight-control software — feeding it the realistic sensor model and letting ArduCopter and ArduRover fly the setpoints.

16 out of 16 collision-free, both vehicle classes, eight scenarios each. The quad passed immediately. The rover clipped obstacles until we found the cause — we were launching it as a car (Ackermann steering, fixed turn radius) when the controller assumes a skid-steer that turns in place. A vehicle-model mismatch, not a controller fault. With the right model, 8/8.

So do we have IRL L5?

No — and we're not going to say we do. Here is the exact ladder:

• ✅ L5 in idealized sim
• ✅ L5 under realistic deterministic sensing, on the device code path
• ✅ 99.5% collision-free under realistic noisy sensing (800 randomized runs)
• ✅ 16/16 collision-free through the real autopilot in SITL, quad and rover
• ⬜ Actual hardware flight — not started

Everything above the last line is sim and software-in-the-loop. “IRL L5” means the last line: real aircraft, real sensors, real flights, and we have zero of that data. What we can honestly say is: L5 in sim, robust to sensor noise, and validated on the real autopilot — hardware pending. The next milestone is a physical rover on a measured course.

The reason we're writing this the way we are — negative results and all — is that the version where we quietly patched the benchmark and kept the L5 banner would have been the easy one, and the wrong one. The gap between “works in the demo” and “works on the vehicle” is where autonomy programs actually live or die, and the only way through it is to keep finding your own asterisks before the field does.

The full technical write-up — the realistic-sensing benchmark, the failure taxonomy, the SITL results — is in the companion paper: From L5-in-Sim to the Real Autopilot: A Sim-to-Real Case Study. The original L5 result is here.