Blog

AI-generated end-to-end API tests, with Cyrille Martraire and Olivier Penhoat

AI-generated end-to-end API tests, with Cyrille Martraire and Olivier Penhoat

Posted on 2026-11-16 - 4 minute read

Cyrille Martraire and Olivier Penhoat opened their talk with the question everyone wants answered: can AI solve the legacy problem? Their honest answer was no, not fully, legacy is still the final boss. But their DDD Europe talk told the story of a real project where AI made something that used to be prohibitively expensive not just possible, but genuinely good.

Key takeaways

The project: migrating a huge API without breaking anything. Their client, a director of engineering at a mobility ticketing company, needed to move an API worth €500 million a month in revenue onto a new platform. The old and new APIs had completely different JSON contracts, and the new format was still changing weekly. The real question underneath the migration was confidence: the old API was trusted because it had years of production use behind it, the new one had none.

The insight: different contracts, same behaviour, so treat it as translation. Once the team realised the two APIs should behave identically despite looking nothing alike, the problem reframed itself as a translation task, exactly what AI is good at. That reframing made a previously uneconomical approach, comparing every response from both systems in detail, suddenly worth trying.

Big JSON files broke the context window, so they queried instead of reading. Early attempts to feed entire 200 to 500 kilobyte JSON files into an LLM ran straight into the classic "lost in the middle" problem. The fix was to treat the file like a database and query it with an MCP JSON-path tool, pulling out only the data needed rather than loading the whole file into context.

Calling AI live for every test run was too slow, too expensive, and not reliable enough. Their first working version called AI directly for every comparison. It worked, but cost around one euro and four to five minutes per test case, and failed unpredictably around a fifth of the time. For end-to-end tests, that's disqualifying: a test you can't trust isn't a working test.

The fix: use AI to generate the test code, not to run the test. Rather than asking AI to solve each comparison live, they asked it to generate the TypeScript code that performs the comparison. The generated code then runs fast and deterministically, with no AI involved, and AI is only needed again when the API schema changes. Martraire called this the balance they were looking for between cost, speed, and determinism.

When a schema changes, the tests fail in a distinctive way, and that failure triggers self-repair. A schema change doesn't produce a normal red test; it produces a parsing failure, a different failure mode entirely. That specific signal triggers AI to analyse what changed, write the new extraction rules, and regenerate the TypeScript and YAML, after which the tests are repaired and running again, often within minutes rather than the hours a human analysis would take.

Ownership transferred to the QA team faster than expected. The team had worried about handing the tool over would be a slow knowledge-transfer exercise. In practice, a QA team with no coding background was handling the YAML framework within a week and generating new plugins for schema changes within a month, working with the original team for two short sessions a week.

Design philosophy: build the framework by hand, generate the laborious part. Martraire described the approach as a lock: humans design the small, precise framework of interfaces and requirements, and AI generates the much larger volume of code that satisfies it, checked automatically against business rules, commit conventions, and linting until everything passes. Penhoat offered a jazz metaphor for the same idea: AI improvises within a harmony of specifications and against a pulse of checkpoints that can't be skipped. Improvisation is supervised freedom.

This reopens techniques that used to be too expensive to use. The headline realisation for Martraire was that end-to-end tests, long considered fragile and not worth the maintenance cost, became viable again once AI could absorb the laborious parts. He sees the same opening for property-based testing, large-scale refactoring, the Mikado method, and model-based or fuzz testing: techniques QA teams have always wanted but rarely had the capacity to run at scale.

The one thing to try next

Don't ask AI to solve your hardest problem directly. Build the small, well-specified framework yourself, the part you can prove is correct, and let AI generate the large, repetitive part that fits inside it. That boundary, not the AI itself, is what makes the result trustworthy.

Watch this video on YouTube