A new stress test of self-improving AI agents has found that systems designed to learn from experience can become less reliable when the order of their tasks changes. The study, submitted August 18 by Salesforce AI Research, challenges a convenient assumption behind agent memory: that saving lessons from earlier work will steadily improve what the system does next.
The researchers re-evaluated two prominent memory methods, Agent Workflow Memory and ReasoningBank, across three web-navigation test environments. Their central result was not that memory never helps. It was that apparent improvement can depend on hidden features of an evaluation, including which tasks arrive first and how much randomness separates one run from another.
A curriculum hiding in the benchmark
Many agent benchmarks present tasks in a fixed order. In the WebArena tests examined by the Salesforce team, that order tended to move from easier tasks toward harder ones. A memory system could therefore collect useful early successes before confronting more difficult work. The sequence functioned like an unannounced curriculum.
When the researchers shuffled the tasks, the picture changed. ReasoningBank moved from an average gain of 1.5 percentage points under the default WebArena order to a 4.5-point decline under randomized orders. Across the tested settings, performance degraded significantly in six of eight shuffled comparisons.
TENS analysis: The decisive issue is not whether an agent can retain a lesson. It is whether the lesson survives a realistic workload in which requests do not arrive from simple to complex. A fixed benchmark order can make sequence dependence look like durable learning. Randomization separates an agent that truly generalizes from one that benefits from favorable timing.
One score can hide a wide range
The study also repeated identical experiments three times. That matters because language-model sampling, changing website behavior, and the state accumulated in memory can all push later decisions down different paths.
Across 24 comparisons, adding a self-improvement method increased run-to-run variance in 17 cases, or about 71 percent. On one WebArena subset involving multiple sites, the gap between the best and worst ReasoningBank runs reached 10.42 percentage points. On the GitLab subset, the baseline gap was 4.44 points, while ReasoningBank widened it to 7.78 points.
TENS analysis: A single-run score is especially weak evidence for a stateful agent. Randomness does not end after one answer; it can be written into memory and retrieved later. The evaluation unit should therefore be a distribution across complete task sequences, not one polished trajectory or one headline average.
Why the agent learned the wrong lessons
Manual review of the stored memories revealed a practical failure mode. The tested browser environment did not permit API calls, yet the agents sometimes saved advice recommending API-based solutions. In other cases, an agent learned to wait for user confirmation even though the benchmark could not provide it. Plausible guidance became unusable because the memory-writing process lacked important information about the environment.
The researchers also found that ambiguous tasks and evaluator errors could create misleading feedback. A failed attempt might generate a confident but irrelevant lesson, which could then influence later tasks. Adding task rubrics, environment feedback, and clearer instructions during memory construction recovered 31 percent of the performance loss in shuffled settings, but did not eliminate the gap.
TENS analysis: The finding reframes agent memory as a governance surface, not just a storage feature. Teams need to inspect what an agent records, identify which observations produced each lesson, and provide a way to reject or revise bad memories. More memory can compound error as readily as it compounds expertise.
A direct test of earlier optimism
Google Cloud AI Research introduced ReasoningBank as a way to distill useful strategies from both successful and failed experiences. Its ICLR 2026 paper reported gains in web browsing and software engineering, while the official Google Research explanation emphasized better effectiveness and fewer task steps. Agent Workflow Memory, developed by Carnegie Mellon researchers, similarly reported that reusable workflows could improve web navigation.
The Salesforce study does not erase those results. It changes the standard needed to interpret them. The new experiments use a stronger GPT-5-mini baseline, repeat full runs, randomize task order, and supply ground-truth rewards rather than an AI judge. Under that setup, the no-memory baseline reached 54.8 percent on WebArena, while ReasoningBank averaged 56.3 percent. The researchers report a p-value of 0.23 for that 1.5-point difference across three runs, which they describe as limited statistical significance.
TENS analysis: The comparison suggests that memory methods must justify themselves against the current underlying model, not the weaker agent available when the method was introduced. As base models improve, a complicated memory loop may add less capability while still adding variance, cost, and another channel for silent mistakes.
What a stronger evaluation should require
The next generation of agent tests should report repeated end-to-end runs, publish best-to-worst ranges, randomize task order, and document every environmental constraint available to the memory writer. Results should also distinguish ordinary model gains from benefits created by the memory system itself.
The study remains a preprint, uses three runs per setting, and focuses on memory-based web agents. Its authors say significant unexplained gaps remain. Even so, the work identifies a concrete standard for claims about AI systems that improve over time: learning should remain visible when the curriculum disappears.
Sources: Salesforce AI Research; Google Research; the ICLR 2026 ReasoningBank paper; Carnegie Mellon University’s WebArena project; the Agent Workflow Memory research paper.
Featured image: Sierra supercomputer at Lawrence Livermore National Laboratory. Photo: U.S. Department of Energy via Wikimedia Commons. Public domain, United States government work. Center-cropped from 5,445 × 3,630 pixels to 5,445 × 3,063 pixels at 16:9 and resized to 1,920 × 1,080 pixels; no generative or substantive alteration.
