Does task-relevant memory reach the agents that need it — while inapplicable content stays isolated? CoMemBench turns source-grounded multi-agent workflows into matched clean–polluted memory experiments with typed, version-aware artifact handoffs.
Agents form local memory from their own tasks, observations, and execution histories. In a workflow, whether that memory should cross an agent boundary is task-conditioned: a downstream node either requires the artifact, or must stay isolated from it. CoMemBench evaluates exactly that boundary.
Applicability is defined by workflow dependency edges, not by speakers, roles, or privacy policies. A document required by node A is a distractor for node B when no edge connects them.
800 pollution instances are paired with controls. If inapplicable content changes downstream behavior, the matched pair exposes it — without equating benchmark-visible artifacts with internal memory.
Artifacts carry types and versions (repository commits, Lean toolchains, corpus revisions). Verifiers check that what crossed a boundary is what the receiving node declared it needed.
Every family is a directed workflow of typed artifact handoffs, grounded in a real source benchmark and executed with domain-native verification.
Long-horizon application workflows with live tools — search, finance APIs, office documents, messaging. Memory includes tool state, intermediate artifacts, and receipts that later nodes depend on.
Real GitHub issues turned into multi-node workflows: reproduce → localize → patch → test → release. Version-aware handoffs pin commits so a stale analysis cannot silently satisfy a downstream node.
Hard multi-clue questions over a fixed candidate corpus. Agents gather evidence partitions, reconcile entities, and return answers with auditable provenance receipts — often under bounded revision loops.
Real proof-dependency graphs from HepLean. A downstream agent may only receive a successfully compiled ProofReceipt whose theorem ID, statement hash, commit, and toolchain match its declared incoming edge.
From static evidence use toward governed multi-party memory — and from interaction topologies toward executable task dependencies. CoMemBench sits at the intersection: task-conditioned memory boundaries on executable dependency workflows. Click any card to open the paper.
LongBench, RULER hsieh2024ruler and KILT / BEIR established static evidence use. MemoryBank, LoCoMo, LongMemEval, RealTalk, StoryBench moved to persistent and interactive memory; MemoryAgentBench added selective forgetting. GroupMemBench, GateMem, PAC-Bench then asked who may see what.
AgentBench, MindAgent, LLM-Coordination study coordination; MultiAgentBench varies star / chain / tree / graph interaction protocols zhu2025multiagentbench; TaskBench and MAS-Orchestra vary task dependencies — chains, DAGs, depth, breadth, parallelism.
τ-bench, AppWorld, Toolathlon li2025toolathlon and Toolathlon-GYM make success checkable in real environments. PeopleJoin and SILO-BENCH add distributed information. CoMemBench combines these lines with typed handoffs and matched pollution pairs.
Verbatim excerpts from the v1 release (workflow_families.jsonl). The goal is what every agent in the family is ultimately working toward.
Three construction modes, one revision contract, and domain-native execution. Topology is a reporting and stress dimension — not an asserted cause of failure without matched interventions.
Two standard measures and two memory-specific measures. Infrastructure failures are reported separately and never counted as wrong answers.
End-to-end workflow success over the full 50-family domain denominator. Reported as completed / registered.
Macro-mean of native_passed over nodes. Captures whether each node completed its own declared obligation under domain-native checks.
Micro-ratio of verified eligible handoffs — downstream success conditioned on successful mandatory predecessors. Proxy for required-memory availability.
Success on polluted instances conditioned on matched control success. Blank when the denominator is below 10 — never reported as zero.
Three backbone snapshots.
| System | Org. category | Stateful Tool | Repository Code | Offline Retrieval | Formal Mathematics | Macro |
|---|
CoMemBench v1 is published on HuggingFace — 800 complete problems across four domains, each with one matched polluted twin. No held-out split, no gate: download and run.
workflow_families.jsonl (800 problems), node_contexts.jsonl (8975 memory scopes), retrieval_corpora.jsonl & math_cases.jsonl, topology descriptors, the instance indexes, and the JSON schemas.
Plain JSON Lines — or use the datasets loader. Each family is a layered DAG of worker nodes with typed handoffs and a bounded revision contract.
Evaluation oracles and pollution plans stay held out. SR / VNCR / VHS / ICS are computed under the published blanking and denominator rules — infra failures never count as wrong answers.
If you use CoMemBench, please cite the following work. A PDF of the manuscript is at comembench.world/paper/.
@misc{zhao2026comembench,
title = {{CoMemBench}: Benchmarking Collaborative Memory
Boundaries across Multi-Agent Workflow Topologies},
author = {Zhao, Sen and Kong, Ruiqi and Zhang, Zuyu and
Shen, Lifeng and He, Xinyu and Zou, Ding and
Zhang, Xu and Zhang, Qinghua},
year = {2026},
howpublished = {\url{https://comembench.world/paper/}},
note = {Manuscript under review}
}