StreamMemBench

Streaming Memory Benchmark: Stage-Level Diagnosis with Evidence Dependency Control

Code release for the paper Streaming Memory Benchmark: Stage-Level Diagnosis with Evidence Dependency Control, published at the Lifelong Agent Workshop @ ICLR 2026.

Overview

StreamMemBench evaluates memory systems in streaming interaction settings where evidence arrives episode by episode and must be formed, managed, retrieved, and applied later to answer queries.

This public repository focuses on the benchmark runner, protocol implementations, dataset exporters for bundled datasets, and the local adapters that are currently intended to be exercised from this codebase.

Benchmark Protocols

Protocol	What it measures
Protocol-F	Formation quality: whether relevant evidence is extracted and stored after each episode
Protocol-M	Management quality: storage usage and management efficiency
Protocol-R	Retrieval quality: whether relevant memories are returned when queried
Protocol-A	Application quality: end-to-end answer accuracy using the system's retrieved memories

Included Components

Datasets with repo-local streaming exporters:

• locomo
• personamem

Adapters documented for the public release:

• llm_baseline
• memos_local
• mem0_local
• memoryos_local
• evermemos

Runtime notes:

• llm_baseline uses the benchmark code in this repository, but its current LLM wrapper depends on EverOS memory_layer modules.
• mem0_local requires the mem0ai package in the active Python environment.
• memos_local requires the memos runtime in the active Python environment.
• memoryos_local requires the memoryos runtime in the active Python environment.
• evermemos depends on EverMemOS runtime modules such as memory_layer, api_specs, and common_utils, which are not vendored in this public release.

---

Quick Start

Prerequisites

• Python 3.10+
• Local copies of the dataset files you plan to use
• API access for the LLM-backed evaluation paths you want to run

1. Install

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

This installs the core benchmark dependencies only. Optional adapter runtimes are not fully vendored or pinned in requirements.txt.

2. Configure Environment

Copy .env.example to .env.

You do not need to fill every variable.

Minimal configuration for most bundled runs:

• LLM_API_KEY

Common optional settings:

• LLM_BASE_URL
• LLM_MODEL
• LLM_PROVIDER
• STREAMING_DATA_ROOT

If you plan to use local external memory-system runtimes, the relevant path variables are documented in .env.example.

3. Optional external runtimes

Several local adapters rely on upstream repositories that are not vendored here.

Adapter	Upstream source	Typical setup
llm_baseline	EverMind-AI/EverOS	clone EverOS and set EVERMEMOS_SRC=<EverOS>/src, or otherwise make memory_layer importable
evermemos	EverMind-AI/EverOS	clone EverOS and set EVERMEMOS_SRC=<EverOS>/src
memos_local	MemTensor/MemOS	install the MemOS package, or clone the repo and set MEMOS_SRC=<MemOS>/src
memoryos_local	BAI-LAB/MemoryOS	install the package, or point MEMORYOS_SRC at the upstream package path used by your checkout
mem0_local	mem0ai/mem0	install mem0ai, or clone the repo and install it in editable mode

Example clone commands:

git clone https://github.com/EverMind-AI/EverOS
git clone https://github.com/MemTensor/MemOS
git clone https://github.com/BAI-LAB/MemoryOS
git clone https://github.com/mem0ai/mem0

4. Prepare data

This repository does not redistribute benchmark dataset files.

Expected local paths if you use the repository-relative layout:

• data/locomo/locomo10.json
• data/personamem_32k_locomo_style_v3.json

Alternative ways to point the benchmark to local data:

• pass --data /absolute/path/to/file.json
• set STREAMING_DATA_ROOT for LoCoMo and PersonaMem

See data/README.md for the expected filenames and resolution order.

5. Run a bundled baseline

Minimal example without LLM judging:

python run_streaming_eval.py \
  --dataset locomo \
  --adapter llm_baseline \
  --data /absolute/path/to/locomo10.json \
  --max-episodes 1 \
  --no-llm-judge

6. Run a local memory system

python run_streaming_eval.py \
  --dataset locomo \
  --adapter mem0_local \
  --data /absolute/path/to/locomo10.json \
  --max-episodes 1 \
  --no-llm-judge

The local adapters require their corresponding upstream runtimes to be installed separately and made importable in the active environment.

7. Build a custom streaming dataset

python scripts/build_streaming_dataset.py \
  --dataset custom \
  --data /absolute/path/to/your_data.json \
  --output ./output/custom_streaming \
  --sample-index 0

8. Launch the local dataset UI

python scripts/dataset_ui_server.py --port 8765

The UI writes session artifacts under output/ui_sessions/<timestamp>/, which are ignored by git.

Helper Scripts

Task	Command
Interactive evaluation	./run_eval.sh
Single evaluation	./run_eval_easy.sh <adapter> <dataset> <episodes>
Evaluate multiple adapters	./run_eval_loop.sh <dataset> <episodes> [adapter...]
Evaluate a UI-generated session	./scripts/run_eval.sh <session_dir> [adapter] [episodes]

These scripts resolve the repository root dynamically and do not depend on machine-specific checkout paths.

---

Repository Layout

• run_streaming_eval.py — main streaming evaluation entry point
• evaluation/streaming/ — streaming protocols, adapters, runner, and management logic
• evaluation/src/ — shared evaluators and adapter implementations
• data_construction/ — dataset loaders, exporters, and streaming construction pipeline
• scripts/build_streaming_dataset.py — CLI tool for dataset conversion and export
• scripts/dataset_ui_server.py — local UI for authoring and inspecting datasets
• data/ — documentation for expected local dataset filenames
• demo/ — static assets used by the dataset UI

---

Open-Source Scope

The public repository excludes:

• local .env files and API keys
• runtime outputs under output/
• private or third-party dataset files unless redistribution rights are confirmed
• internal agent-instruction files and internal documentation that are not part of the public code release

This repository is a code release, not a fully self-contained artifact. The baseline path is bundled; several local adapter paths depend on separately installed upstream systems.

Current repository provenance:

• public git remote: https://github.com/landian60/StreamMemBench
• local adapter integrations currently reference upstream projects such as EverOS, MemOS, MemoryOS, and mem0, which should be obtained from their respective upstream repositories if you want to run those paths

---

Citation

If you use StreamMemBench in academic work, please cite the workshop paper.

BibTeX is provided in CITATION.bib and repeated here for convenience:

@inproceedings{liu2026streaming,
  title={Streaming Memory Benchmark: Stage-Level Diagnosis with Evidence Dependency Control},
  author={Liu, Guanming and Yin, Haoran and Li, Tianchen and Yan, Sikuan and Wang, Hongru and Chen, Baian and Ma, Xiaoteng},
  booktitle={Lifelong Agent Workshop at ICLR 2026},
  year={2026}
}

---

License

This project is licensed under the MIT License.