More and more systems bolt memory onto agents, and every one of them claims it gets smarter the more it is used. But one question is rarely answered head-on: context windows are growing too. If a model can read more and more in a single pass, how much is pre-stored memory still worth?

EvoMemBench happens to put a number on that curve. It places 15 memory methods under one evaluation protocol and reports: as the context budget grows from 16K to 128K, the best gain of memory methods over the no-memory baseline shrinks from +14.5 points to +8.5 points — and at the 128K tier, some memory methods have already dropped below the no-memory baseline.

What I care about is whether that curve still holds with a different backbone — especially the cheapest tier. The reason is practical: at this kind of experimental scale (the 128K tier means replaying long contexts over and over), flagship pricing simply cannot carry the run, so the backbone gets filtered by price before anything else; and in the deployment settings I care about, the volume also runs mostly on cheap tiers. So I did not replicate the paper’s full setup. Instead, I pre-registered a reduced question: on the cheapest GPT tier obtainable from the gateway, in a single execution environment, does the direction — gains shrinking as the budget grows — still hold?

The headline first: the direction holds. The best gain I measured shrank from +18.0 points at 16K to +6.0 points at 128K, with both memory methods narrowing in the same direction. But the paper’s sharpest scene — memory dipping below baseline at 128K — did not appear: the retrieval memory’s gain merely went to zero, and the procedural memory stayed above baseline. The composite verdict is PARTIAL; the evidence boundaries are at the end of this article.

First, the evidence grades

This paper (arXiv:2605.18421, from HKUST (Guangzhou) and others) released v1 in May 2026 and was updated to v3 on October 30 — it is not a new paper. What it does is set up a coordinate system for the question “does an agent’s memory actually help?”: one axis is the scope of memory (within a single task / across tasks), the other is the content type (knowledge / execution), forming four quadrants; six datasets, 2,954 evaluation samples in total per Table 2, fall into the four quadrants; 15 memory methods are grouped into six families and compared against long-context baselines under a unified protocol. One condition must stay attached at all times: every memory method in the paper shares a single backbone, DeepSeek-V3.2; its ranking is a ranking under that backbone, and talking about rankings detached from the backbone is meaningless.

Of the paper’s eight findings, three in full — plus a half — bear directly on this article. The long-context baseline is still very strong, leading on knowledge retention and revision (Finding 1); memory’s bottleneck is revision rather than retrieval — the methods are good at storing information, not at revising old knowledge when the evidence changes (Finding 2); one layer deeper, the bottleneck is forming reusable knowledge from what is stored, not merely storing more (Finding 6); procedural memory fits execution tasks best (Finding 4); and there is a difficulty flip: on easy tasks memory is a negative gain, while on hard tasks only memory scores at all (Finding 5) — that half I could not test, for reasons given at the end.

Everything above is author-reported. My replication touches exactly one of these: Finding 3’s gain-versus-budget curve, and only in one environment of one quadrant.

My replication design: the cheapest backbone, one curve only

This experiment was pre-registered; every setting was fixed before the run started, as follows. The subset is InEp-Exec from the four quadrants (execution tasks within a single episode), running only one of its environments, Gorilla File System — a set of multi-turn tool-calling tasks whose success is judged by the checker built into the official runner, with no extra judge needed. Three arms: A0, the no-memory baseline; A1, BM25 retrieval memory; A2, the procedural memory ACE. Two context-budget tiers: 16K and 128K. Fifty samples per cell, 300 episodes in total, with the official protocol parameters (function-calling mode, long-context noise, memory scope) taken as-is from the official scripts.

The backbone is the endpoint labelled gpt-6-luna at a third-party OpenAI-compatible gateway, fixed for the entire run. One identity boundary has to be stated up front: the label is the gateway’s; the model it actually serves cannot be independently verified from my side. So every number of mine below holds only for “this labelled endpoint” — a variant caliber of the paper’s DeepSeek caliber, not the same table.

Deviations from the paper’s setup, and version details, were written into the experiment plan before the run; three things to disclose:

• Sample count: v3 matches the repository at 50 per environment. Table 2 of this paper’s v1 said 200 per environment; v3 changed it to 50, and the official repository ships 50 as well — those 50 are what I ran. This also means 50 per environment is the paper’s own evaluation scale: the ±14-point statistical intervals later in this article sit on the paper’s per-environment numbers just as much.
• The procedural memory implementation was swapped. The plan originally specified ReasoningBank, but the implementation shipped with the repository hard-depends on another vendor’s embedding key, and the gateway is my only access route — so I switched to ACE, which belongs to the same procedural family and is the paper’s other headline method. It is not the same method: in the verdict it can only be read at the family level, not matched method-to-method.
• The memory arms got a different transport layer. The calling channel the authors prepared for the memory methods is another cloud vendor’s batch interface, whose OpenAI-compatible backend in turn rejects memory types; in my local working copy I replaced only the transport layer — protocol, prompts, memory logic, truncation and scoring all use the pinned official code. One upstream asymmetry is worth disclosing while I am here: the authors’ OpenAI backend passes temperature 1 on the A0 arm’s calls, while the path the memory arms take passes no such parameter — I kept it exactly as-is rather than editing code to align it, so between the three arms there is, besides memory itself, this layer of generation-parameter difference.

Results: the shrinkage replicated; the dip below baseline did not

The six cells’ success rates are as follows (n=50 per cell, official success criteria):

Arm	16K	128K	Gain (vs same-tier A0)
A0 no memory	38% (19/50)	44% (22/50)	—
A1 BM25 retrieval	42% (21/50)	44% (22/50)	+4.0 → 0.0
A2 ACE procedural	56% (28/50)	50% (25/50)	+18.0 → +6.0

Both memories’ gains narrow as the budget grows; the best gain shrinks from +18.0 to +6.0. Under its DeepSeek caliber, across four environments, the paper reports +14.5 → +8.5 (and not monotonically in between: the 64K tier dipped to +7.8 before the 128K tier rose back to +8.5). Same order of magnitude, same direction — but these two sets of numbers are not cells of the same table and cannot be compared side by side for size.

The 128K tier’s details deserve a separate word. The paper says that at this tier some methods have already dipped below baseline; in my run, none did. BM25’s gain evaporates to exactly zero (44% against 44%), stopping at the threshold of “dipping below” without stepping over it; ACE remains 6 points above baseline. This result neither confirms nor refutes the paper’s clause: I tested only 2 of the 15 methods and 1 of the 4 environments, and the two I picked sit at the ends of the spectrum — the simplest retrieval memory, and the strongest class of procedural memory in the paper’s execution quadrant; the methods the paper names as dipping below at 128K, MemAgent and MemoBrain, are not among them, so part of “no dip” is decided by the selection itself. What can be said is that, on this subset and this backbone, “memory becomes a negative asset at 128K” did not happen; what happened instead is “the cheap memory fails first” — the retrieval memory’s gain is the most budget-sensitive, the procedural memory’s gain more durable. Take ACE’s 12-point shrinkage apart: half comes from the baseline itself getting stronger (A0 got 3 more tasks right), the other half from ACE falling back (3 fewer) — both movements are only 3 tasks in size; together they make a picture, taken apart they make no story. This cell also points the same way as the paper’s Finding 4: ACE, the only procedural method among the three arms, is the strongest at both tiers.

Now a statistical reality check: each cell has only 50 samples, so one cell’s 95% interval is roughly ±14 points, and a 4-point gap is barely two tasks. Fortunately, all six cells ran the same batch of 50 tasks, so the flips can be read paired: at the 16K tier, ACE got 9 more tasks right than the baseline and 0 fewer; but take the shrinkage magnitude itself to a paired bootstrap and the lower bound of ACE’s 12-point 95% interval sits exactly on 0, while BM25’s shrinkage is indistinguishable from zero. So this article’s directional verdict is descriptive: the picture points the same way as the paper’s, with the ACE arm backed by paired evidence; arguing wins and losses off any single cell is finding conclusions in noise.

The rent memory has to pay: a cost anatomy

The paper reports scores, and it also reports each method’s total tokens — but that is a combined total of task execution and the memory module (see its appendix efficiency tables). It does not break out the memory module’s own calls separately. This time I split them by purpose, and got a set of numbers the paper does not have.

The 300 episodes consumed about 15.17M input tokens and about 540K output tokens in total. Across its own 100 episodes, ACE’s memory module — the reflection and experience-curation calls after each round — consumed 2.66M tokens: counted against ACE’s own input alone, the memory module accounts for about 36.5%; spread over the whole 300-episode run’s total input, it is 17.6%. Its output tokens run about 7.4× the no-memory arm’s, because reflection and curation are both generative calls; and since the unit price of output is 6× that of input, in money terms that 7.4× hurts the bill more than the input-side share. In other words, for every 100 input tokens ACE spends, about 36.5 went to maintaining memory rather than advancing the task itself. BM25 produces no memory-maintenance calls: indexing and retrieval are done locally and never call the model; but the retrieved text still has to enter the context, and that input is booked to its own task account — what it saves is the maintenance calls, not the context.

For a sense of magnitude, compare against the paper’s own efficiency table: under the DeepSeek caliber at the 128K tier, ACE’s total tokens run about 1.8× the no-memory baseline’s, and the excess accounts for about 43.5% of ACE’s total — the same order of magnitude as the 36.5% I split out here (different caliber: the paper’s figure is a ratio of totals against the baseline; mine is a by-purpose split inside ACE’s input).

These numbers change the math on whether memory is worth it. A +18-point gain is not free: ACE alone pays rent of more than a third of its own input, plus over seven times the output volume; and that rent exists regardless of the budget tier — when the gain shrinks to +6 points at the 128K tier, ACE’s memory module still consumes almost the same number of tokens (1.34M at the 16K tier, 1.32M at the 128K tier). When evaluating a memory method, the memory’s own calls should be counted as part of the method’s cost, not waved through as infrastructure. Converted at the Luna tier’s public list prices (input $0.20, output $1.20, both per million tokens), the total price of these 300 tasks comes to about $3.7 (the gateway account’s actual billing is the gateway’s own, which I cannot see) — what’s cheap is the backbone; what’s expensive is the structure.

What I didn’t test

• Finding 5 (the difficulty flip) was not tested. The pre-registration left a secondary probe, using the authors’ own difficulty tiers to test “memory is negative on easy tasks, positive on hard tasks” in the CrossEp-Know quadrant. The probe reached the pilot stage, the gateway account’s quota ran out, and I stopped it; no readable data was obtained. So this article draws no conclusion whatsoever about Finding 5; where it appears above, it is strictly author-reported.
• The paper’s long-context comparisons (proprietary models such as Gemini-3-Flash and GPT-5-mini) cannot be re-run by me; their scores are cited as author-reported.
• The other three quadrants, the other thirteen methods, and the paper’s original DeepSeek-V3.2 caliber are all outside this article’s evidence scope.

This connects directly with this blog’s earlier Beacon cross-tool memory field test. That one’s conclusion was: memory adds value only for “knowledge the repository itself cannot derive”, and the automatic capture and distillation layers are fragile. The two pieces of evidence take different shapes — that one dug deep into a single system, this one is a subset replication under a cross-comparison protocol — but they point at the same thing: memory’s gain never exists by default; it is a function of a set of conditions. The condition this run adds is the budget: the cheaper and larger the context, the less pre-stored memory is worth; and the first to fail is the kind that only retrieves.

Conclusions

If you are choosing a memory scheme for an agent, this replication can support only three uses. First, state the budget before quoting the gain: for any improvement a memory method claims, ask at what context budget it was measured — scale the budget up one tier and the gain may evaporate by more than half. Second, write the memory module’s own tokens into the cost table: the rent of generative memory (reflection, curation, summarization) can reach more than a third of its own input (36.5% in this case), and it does not shrink as the gain shrinks. Third, on this kind of tool-calling task, procedural memory (ACE in this case) is stronger at both tiers, but you pay for its calls; pure retrieval memory’s gain hits zero first when the budget is ample, and it fits only scenarios where the budget is genuinely constrained. Don’t rush to carry this out of this environment: the paper’s Finding 7 shows that other execution domains, such as web search and embodied tasks, have different best-fitting memory forms.

As for the benchmark itself, my verdict is this: Finding 3’s shrinkage pattern, moved to a different endpoint and shrunk to a subset, still points in the same direction — that is its most useful part; but the endpoint’s identity cannot be independently verified, so this counts as a cross-endpoint exploratory replication, not a cross-backbone proof. The sharper claim — dipping below baseline at 128K — did not appear in my subset; judging whether it holds for more methods will take someone running the remaining thirteen methods under the original caliber. That is not this article’s conclusion; it is the open question this article leaves for future work.

---

Evidence boundaries: all paper numbers are author-reported from arXiv:2605.18421 v3; my numbers hold only for the subset above, the three arms and two tiers, and the gateway-labelled endpoint, in a single run. The full receipt (including the six cells’ raw counts, the token split, the deviation log and the statistical limitations), the per-task data and the paired-analysis scripts are published with this article (see publicRepository at the top) and can be verified file by file against SHA256SUMS.