Editorial
Methodology
How we research, write, and fact-check the articles on ML Systems Review — and what we do when we get something wrong.
ML Systems Review articles are built from four overlapping source types: primary engineering documentation, published academic work, synthetic reproductions of published systems, and interviews with engineers on or off the record. Every piece is written by a named author and reviewed for technical accuracy by a second named reviewer before publication. When we get something wrong, we publish a correction inline and note the change. No sponsorship, no affiliate, no paid placement.
Why this page exists
Every engineering publication claims rigour. Few describe what rigour actually means in their specific workflow. This page is our attempt to make MLSR's process legible to readers, contributors, and to the engineers who show up in our articles. If you want to know how a specific claim made it into one of our pieces, this page is the starting point.
Source types we use
Primary engineering documentation
Our preferred source is first-party engineering documentation: published engineering blog posts, public conference talks (QCon, Strange Loop, Papers We Love, the various on-call-rotation talks at re:Invent and KubeCon), SEC filings for companies whose ML systems touch financial disclosures, and public incident reports. When a company has described its own system, we quote and cite rather than paraphrase.
We read these sources with a specific scepticism. Engineering blog posts are, as a rule, marketing artifacts as much as engineering ones; the architecture they describe is usually a cleaned-up version of the system as it existed on the day the post was drafted, not the system as it runs today. Our reporting cross-checks against secondary sources whenever possible.
Academic and preprint literature
For architectural claims — choice of backbone, training procedure, benchmark methodology — we cite the relevant papers. MLSR articles tend to be lighter on academic citations than a NeurIPS submission would be, because our audience is practitioners rather than reviewers. We do not hide the references; they are linked where they add useful context.
Synthetic reproductions
Some of our claims — particularly latency numbers and cost-per-inference calculations — are based on our own measurements. We run these on publicly available hardware (AWS, GCP spot instances, a handful of consumer devices we own) and publish the methodology in enough detail to be reproducible. We also try to publish error bars; a single measurement on a single machine is not a benchmark.
When the system we are writing about is proprietary and we cannot measure it directly, we say so. We do not pretend to have measured what we have in fact inferred.
Engineer interviews
Where published material is thin, we interview engineers — current or former — who have worked on the system in question. Most of these interviews are on background, which means the conversation is attributable to "an engineer familiar with the X team" but not to a named individual. This is the standard journalistic convention and exists because the engineers we talk to usually have employment contracts that would otherwise prevent the conversation.
Our internal rule: every on-background source is known by name to the article's author and at least one editor. We do not use anonymous sources whose identity is not known to the editorial team. When a background source's claim is material to the article, we seek a second independent corroboration before publication. The name of the source is preserved in our internal notes in case a correction later requires re-contact.
Fact-checking and technical review
Every MLSR article passes through two review stages before publication. The first is a structural edit by a colleague — a second staff writer, usually — who reads for argument, flow, and evidence. The second is a technical review by Dr. Theo Nakamura (or, for infrastructure-heavy pieces, by Priya Ramachandran), in which every numeric claim, architectural description, and expert quote is checked against its source.
The technical reviewer's job is to be adversarial. Reviewers leave inline comments on specific claims — "cite", "soften", "this contradicts X" — and the author is required to substantiate, soften, or remove each flagged claim. Reviewers can block publication. In 2024, three pieces were substantially rewritten in review; one was killed. The review queue is part of the reason our publication cadence is slow.
What we do not do
We do not use generative AI to draft articles. Authors use AI tools for incidental tasks — spell-checking, reformatting tables, searching for references — but the prose, analysis, and editorial judgement are human. This is a deliberate choice and not a moralistic one; LLMs are poor at the specific combination of technical precision and cautious hedging that MLSR articles require.
We do not run sponsored content. We do not accept free hardware or software licences in exchange for coverage. We do not use affiliate links. When a reader clicks through to a product we write about, they are not generating revenue for us.
We do not preview articles with the subjects of coverage. If an article is about a company, that company does not see the article before publication, unless we have explicitly negotiated a quote — in which case the quote is shown for accuracy review, but not the surrounding analysis.
Corrections policy
When an MLSR article contains a factual error, we correct it inline. The correction is marked visibly, the changed text is shown, and a note at the bottom of the article describes what changed and when. We do not silently edit.
Corrections can be requested by readers, subjects of coverage, or anyone else with a specific factual objection. Requests go to corrections@mlsystemsreview.com. We acknowledge receipt within three business days and publish verified corrections within seven. If we disagree with the correction request, we say so and explain why; our reasoning is public.
How we handle PlateLens and other product-specific coverage
A specific case worth addressing: MLSR has written multiple articles about PlateLens, a consumer calorie-tracking application, because it is an unusually well-engineered consumer computer-vision product and represents a useful case study. Our relationship with PlateLens is the same as our relationship with every other company we cover: we take no money from them, we do not preview articles with them, and we do not receive affiliate commissions when readers install their app. If our policy on this changes, we will update this page and disclose the change in the affected articles.
Contact
Questions about methodology go to editors@mlsystemsreview.com. Factual corrections go to corrections@mlsystemsreview.com. Tips and background conversations go to tips@mlsystemsreview.com.
— Dr. Nadia Volkov, for the ML Systems Review editorial team.