A migration nobody read line by line

The job as it was handed over

A large body of legacy code had to move to a new way of doing things. The default plan was the one every migration starts with: open each file, find the pattern, change it, move on. Around 140 classes and 300 methods.

That plan has a property worth naming. Its output is a reviewed codebase and nothing else. The next time the same pattern appears, in a file written next month by someone who was not in the room, the work starts over, and the only thing standing between the codebase and the old pattern is that somebody remembers.

Writing the check instead of doing the check

So I wrote the migration as ESLint rules. The rule walks the syntax tree and decides whether a given construct is the old shape or the new one. What was a reading task became a program, and the program has the property the reading task did not: it runs again.

This is not a clever idea. It is the obvious idea, and it is worth writing down only because of what stopped it from working.

The linter could not see the runtime

The platform runs code inside its own execution environment. Objects, and the methods hanging off them, are put in scope by the framework rather than imported by the file that uses them. A developer reading the file knows they are there. A linter reading the file does not: it sees an identifier that resolves to nothing.

Point a normal ruleset at code like that and it reports thousands of problems, nearly all of them false. That result is worse than no linter at all, and the reason is not the wasted afternoon. It is that a team which sees a wall of red once learns that the red means nothing, and the next warning, the real one, is scrolled past with the same reflex. A check nobody believes has negative value.

So the work was not the rules. The work was recreating enough of the framework's runtime context, in polyfills, for the static analysis to be correct inside it. Give the linter a model of what the environment provides and the false positives stop being false: what is left is genuinely unresolved.

What it saved

My own team's hand review was replaced by a command. I put that at more than fifty developer hours. The rules were then picked up by other teams facing the same migration, where I would put the saving at a hundred hours at the very least, and I am deliberately quoting the low end of what I believe.

Neither number was measured. Both are my estimate, from the size of the surface and what review of that surface costs, and I would rather label them than round them up.

The part that outlived the migration

The migration finished and the rules stayed. That is the outcome I would actually argue for.

The class of defect they catch used to surface at runtime, which in practice meant it surfaced in front of someone using the product. Now it surfaces in the editor, while the developer who is causing it is still looking at the line that causes it. Same defect, moved from the most expensive moment in its life to the cheapest.

Estimated, not measured

The hours are an estimate and always will be. Measuring them properly would have meant migrating a sample by hand to get a per-class cost, which is spending the thing being saved in order to price it. I chose not to, and the number carries that caveat wherever it goes, including on my CV.

I never counted the defects the rules caught. That is the number I would want, because it converts the story from time saved into quality, and it is gone: the rules were not instrumented, and the runs that mattered are long past.

And the largest effect cannot be counted at all. Once the rule is on, the defect it prevents never happens, so there is nothing to count. Any figure I offered there would be invented.

What carries over

When a task is large and mechanical, the question is not how to get through it. It is whether the judgement being applied can be written down as a rule, because if it can, the migration and the guard against the next one are the same artefact. And when the rule cannot see what it needs, the fix is to give it a model of the environment rather than to loosen the rule until the noise stops.