Regenerating Ablated Code

Delete a function from your codebase and ask a model to write it back. How much of the original returns — and what decided that?

In biology this is a regeneration assay: cut a limb off an axolotl and the surrounding tissue carries a blueprint of what's missing ⁠¹. We wanted to know how much of that blueprint a codebase carries about itself.

In general the regenerated code honors the function's signature and return contract, but content is frequently omitted. And the failures are not independent guesses: code that LLMs are uncertain about tends to regenerate into structurally similar shapes across the models we ran greedily.

Previously we found that comments steer agents on SWE-bench; this work expands that line of research. Future posts will cover how comment inclusion assists recovery of ablated functions, agent trajectory analysis, and common failure modes in regenerated code. Especially of interest is how LLMs rewrite ambiguous, but safety-relevant, code after an ablation and how much the surrounding code impacts those choices.

#What we did

We translate Python code into an abstract syntax tree (AST), breaking down functions to ablate their body and, occasionally, the signature as well. We then feed context from either a tight window of surrounding lines (±40 in the retrieval-mode sweep, ±5 in the origin and convergence runs), the whole file, grep results for the function name, or AST neighborhoods.

Then we examine the output against the structure of the original code to quickly detect differences in implementation. Comparing this against the original code provides us with an easy measure of omitted and excess code written by the LLM. A naive comparison is quite harsh; we use a longest-common-subsequence alignment for a more accurate measure. Two metrics matter here:

• Recall — of the nodes the original had, how many the LLM recovered
• Precision — of the nodes the model wrote, how many match the original

By using these values we can make inferences about the types of errors models make while reconstructing codebases.

We summarize each snippet with a reconstruction score: the harmonic mean of precision and recall (F1).⁠¹

We then check the score against an independent reader: a judging model (GPT-5.2), blinded to the metric and given the original as ground truth, reads 40 low-scoring snippets (reconstruction score below 0.2) and 40 high-scoring controls (0.7 and above), balanced across code origins and shuffled together. Every one of the 40 low scorers was confirmed as losing behaviorally load-bearing code. The reverse direction is weaker: about half of the high-scoring controls still carried a subtler load-bearing change, usually a substituted value, renamed key, or altered message that identifier-anonymized structural overlap cannot see. So a low score reliably means real loss, while a high score is not a certificate of equivalence.⁠²

The judge is still a model reading code. We anchored the score once more, this time against an instrument nothing can argue with: real test suites. On 182 functions from SWE-bench Verified, each one the function a human bug fix touched, regenerated from the surrounding codebase and spliced back, we ran the project's own tests in the official harness. The score comes back calibrated: the odds that a regeneration passes every suite roughly double for each +0.1 of reconstruction score, from about 2% at a score of 0.2 to about 83% at 1.0 (Spearman ρ = 0.36, permutation p < 0.001). No regeneration scoring below 0.2 passed, and the lines the metric said were dropped were the lines whose tests failed.⁠³

The asymmetry survives execution too: a low score is close to a guarantee of broken behavior, while a high score improves the odds without certifying them. One regeneration reproduced 98 of a fix's 100 lines and still failed, because one of the two lines it dropped was an import.

We vary two conditions: code origin — AI-generated (AI-Village) vs. human-generated (pre-2021) — and the amount of context provided. We also examine how much the models agree with one another.

#How well do codebases imply their own functions?

In the following figure each dot represents a unique codebase, either AI or human generated, with the lines showing the interquartile region for a given codebase. Clearly there is some difference between the codebases that easily offer up a world model to an agent and those that do not. At first glance, though, the difference does not appear to be whether the codebase originated from agents or humans.

About a fifth of the function-to-function variance in reconstruction is explained by which codebase the function lives in (ICC ≈ 0.18–0.22 within each origin). Even after adjusting every function for its length and branching complexity, the between-codebase share is unchanged (ICC ≈ 0.21 on the residuals): membership carries signal beyond the kind of functions a repo happens to contain. At the function level the score tracks simplicity: reconstruction falls with branching complexity (ρ = −0.27) and body length (ρ = −0.25), while docstring and type coverage barely register. Popularity shows no sign of the memorization signature: per-repo score against GitHub stars is ρ = −0.17 (95% CI [−0.44, +0.14], n = 50).

Of note, the prior figure is focused on feeding predefined context to a model such as the home file, a grep of the function name, and any file it appeared in. Of the methods we tested, the best result came from finding all references and adding any file they appear in to the context — a brute-force approach to getting the most data to the models.

Meanwhile the AST neighborhood — a more restrained method of gathering context that only includes code within the enclosing def of each call site — performed about as well as dumping the whole file the original function appeared in. Strangely, this did not seem to help Kimi-K2, an avenue for future analysis.

Context is also an equal-opportunity lever with respect to who wrote the code. In a paired comparison on the balanced corpus — every function regenerated with local lines, the whole file, and reference sites — the recall gain from going local → references is +0.26 for AI-built code and +0.24 for human code.

#Failure modes in reconstruction

There are many cases where a model is unable to appropriately recreate the original function. How the dysfunction manifests shows up in the precision−recall gap: a positive delta indicates the code was largely correctly laid out but parts were omitted, while a negative delta indicates the agent wrote more code that was generally incorrect.

There is a peak at around +0.1, but in the GPT-5 models a second hump at around −0.2 appears as well, as if one failure mode were being exchanged for another.⁠⁴ The GPT-5 models also run in a different sampling regime — they reject greedy decoding, so they are a single stochastic draw at temperature 1 with minimal reasoning effort — which means their curves should be read for shape rather than height.⁠⁵ The hump itself, however, belongs to the model family rather than the regime: re-running GPT-5-mini at medium reasoning effort deepens the expansion hump instead of removing it (mean precision−recall gap −0.09 → −0.13), and re-running DeepSeek in GPT-5's sampling regime (temperature 1, single draw) leaves its omission lean intact.

#Models tend to regenerate into a similar shape

Between the three temperature-0 models we noted that the regenerated code matches the other models' output more closely than it does the original code, as measured by the same structural F1.⁠⁶

While the relationship is modest — 0.595 vs. 0.542 — the gap of +0.053 (95% CI [0.044, 0.061]) is significant. Further study into related model families and the exact nature of these patterns might give insight into more obvious correlations in behaviour. On their own they are minor, but even a small repeated bias can snowball into massive changes.

The shared shape is strongest exactly where the codebase says the least. Vary how much context the models see and the attractor moves: with only a few lines around the stub, the models agree with each other far more than with the author (a gap of +0.10 F1). Hand them the whole file, then the referencing files, and both agreements rise — but agreement with the author rises faster, shrinking the gap to +0.06 and then +0.05. Rich context makes the author the shared attractor; sparse context leaves the models alone with their common idiom. It never reaches zero, though: even at the best retrieval mode, the models still resemble each other more than the code they were rebuilding.

#It's a property of the codebase, not the model

The propensity for regenerated code to be similar between models indicates that regeneration is a property of the function, but we wanted to know if this is also true at the level of codebases. If it is, then we have support for the notion that codebases carry implicit knowledge of themselves at varying levels, with some being over- or under-determined.

Ranking the codebases makes the difference clear. We ran five models from three labs — DeepSeek-V3.2, Kimi-K2, Mistral-Large-3, GPT-5-mini, and GPT-5.2 — and every one of the ten pairings agrees on the ordering (ρ between 0.89 and 0.95). The correlation never drops into "different model, different codebases" territory, and it survives the greedy-to-stochastic boundary.⁠⁷

The mental model this leaves is: the model sets the level, the codebase sets the ranking. Each model slides the whole distribution up or down by a few points while which codebases sit at the top and the bottom stays fixed. A codebase that regenerates poorly under one model is likely to regenerate poorly under all five.

That invariance is what makes the property worth measuring at all. A map of which functions your codebase does and doesn't determine keeps its meaning as the models underneath it churn. It's a fact about the code, not the latest model.

#Conclusion

The ability of codebases to regenerate depends on their internal structure, and here we establish that the property is robust across models: models produce similar codebase rankings even when their absolute regenerative capacities differ. Furthermore, as a model accrues context, the regenerated results pull the models toward the author and away from their shared idioms — an indicator of codebases implicitly providing self-models to LLMs that are then used during standard workaday tasks.

This motivates future research into how agents' interpretations of a codebase modify their behaviour during coding tasks and multi-agent environments, as well as the development of codebase interpretability techniques as a means to understand not only the current functional landscape of a codebase but its implied future directions.

We are already working on fleshing out the analysis and results to cover more properties.

Part of our research into codebase alignment. Get in touch if you're interested in collaborating.