The model I'm working with is 100 times smaller than yours, and my AI agent still works great

A few months ago, I asked an AI agent I'm building to analyze a large repo. It started promisingly — it read files, mapped dependencies, for a moment it looked exactly like the demo they run just before a funding round — and then it choked. I stopped the run and looked at the numbers: the analysis required 633,000 tokens, into a context window of 131,000. I ran it on two more repos, just to be sure: 287,000. 513,000. Twice to five times the window, every time. Impressive consistency, just not the kind you upload to social media.

And here's a small confession: a window of 131,000 is not a theoretical choice. The model writing code for me is Qwen2.5-Coder-32B — an open model running locally for me, on a single A100 — orders of magnitude smaller than the models you talk to via API. When the model is small, you don't have the privilege of hoping it will "just work out." Every weakness of it meets you in the face, quickly, and without a Customer Success team to cut corners. And the first weakness is fourth-grade arithmetic. An agent accumulates context with every action — every file read, every search result, every terminal output pile up in the window — and the task doesn't commit to any size.

My first instinct was that of an engineer: to compress. I built a mechanism that shrinks every file read — a skeleton of declarations plus a summary instead of the full content — a saving of five to twenty times per read. The agent crashed again, a little later. Because compression improves the constant, and the problem grows linearly. And no, a million tokens don't solve this. I can already hear the first response: "That's what models with a million-token window are for today." True. Let's still do the math that the slide skipped. A context window, as large as it may be, is a constant. An agent's context grows with every step, and the tasks we give agents grow even faster — a larger repo, a longer investigation, a day's task instead of an hour. A million tokens move the wall eight times further. Impressive. A medium-sized corporate monorepo eats this gap by the afternoon meeting.

Any finite size loses to something that grows. A larger window is the same deal, with a fatter invoice. And that's even before the two walls you hit earlier:

The quality wall: the effective window of a small model is smaller than the window in the presentation. The big numbers are measured in "needle in a haystack" tests — the model pulls one sentence out of a novel, everyone applauds. Ask it to say something smart about the whole novel, and you'll get a student who only read the summary.
The economy wall: an agent drags its context anew with every step, and a serious task is hundreds of steps. Two hundred steps times a fat context — someone pays this invoice at the end.

And there is one hint from the real world that is worth more than all the benchmarks: no engineer keeps an entire monorepo in their head. Human working memory is terrifyingly tiny — and humans navigate codebases of tens of millions of lines, every day, without complaining about the window size. They have a method: a mental map, breaking down into sub-problems, and passing half the work to someone else. Therefore, it is not the model's "fault," and upgrading the model won't save you. It's the architecture.

Change the algorithm, not the bucket

The solution to the 633K wall was to change the way of running. The main agent stopped reading files itself: it splits the investigation between sub-agents, each running with its own fresh context window, digging as much as needed — and returning to the father a summary of four thousand tokens. Division of labor. In another world, they call it "management" and get options for it.

But the math changes fundamentally: 200 file reads in one agent — 300,000 tokens, crash. Five sub-agents, each reading forty files and returning a summary — twenty thousand tokens at the father's. The main agent's context is blocked by the number of sub-agents, not by the number of files in the repo. Asymptotic change, not a constant improvement — and completely indifferent to the window size. With 131,000 or with a million, it works the same way. The wall simply stops being relevant, without anyone upgrading anything.

And this, by the way, is the only benchmark I truly trust: the same model, the same task, with the lever and without it. The three repos from the intro — those that required 287, 513, and 633 thousand tokens and crashed — pass today from end to end, with the exact same model. And for those who want a number from a source that isn't me: Anthropic published that moving to a multi-agent architecture improved their research task performance by more than 90 percent compared to a single agent, on the same models.

Casting instead of prompt

Is the model blind? Put glasses on it. The model writing code for me is completely textual. A user who attached a screenshot of a visual bug — a broken button, a layout that fell apart — previously received exactly nothing. What I did instead: I connected a tiny, separate vision model — Qwen2.5-VL-3B, ten times smaller than the code model — as a tool. It receives the image and returns a structured description — layout, elements, status, immediate suspects — and the textual programmer debugs the UI from the text. A model that doesn't see fixes visual bugs.

Two limitations that no prompt fixes, and the smaller the model, the sharper they are: it has no sense of completion, and it has no self-criticism. The solution is casting. The exact same weights run for me in several roles that the plumbing directs: an executor who writes the code; a verifier who ensures it runs in a completely clean context; and in critical tasks also a skeptic, who gets up in the morning to find flaws. And the decision "we're done" has left the model's hands entirely — no approval from the verifier, no completion. A model, like a PhD student, never submits if you don't take the work from them by force.

And alongside the casting, feedback from reality: after every file edit, the plumbing pushes the compiler errors to the model without it asking. An agent that gets the red in the face after every step corrects itself; an agent that works blind submits broken code with full confidence.

Memory is a retrieval problem

A long conversation must be compressed at some stage — the window, as mentioned, is finite. The accepted solution in the industry: summarize the history and throw away the source. But a summary keeps the spirit and kills exactly what you will need later — the exact path, the variable name, the error message word for word. So I reversed the direction: nothing is deleted. The history goes to the archive, the summary enters the context, and the model gets a search tool for everything that was cut. And a small detail that summarizes the whole article for me: in the first version, the model didn't use the search tool. Ever. What fixed it was one sentence I added to the summary itself: "Full details are available via the search tool." One line of context in the right place, and the ability that was already there came to life.

Weights are a commodity, architecture is cumulative. Every six months, a "game-changing model" comes out, and every January is declared again as the "year of the agents." Meanwhile, notice what is common to everything I described: no lever requires a smarter model, and none of them are thrown in the trash when a new model comes out. A new model raises the floor; architecture raises the ceiling. And the investment in it, unlike a model that changes every six months, is cumulative. So when a new model comes out, I naturally download and check, like everyone else — my addiction is no different from yours, I just pay for it in electricity. But I no longer ask "how smart is it." I ask how far my plumbing will take it.

Yehuda Neuman is the CTO and Chief Architect of PAIS.