Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions benchmarks/experiments/PAGE-SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,7 @@ What belongs beside the pinned ratio: the view's cost per session is real. Its b
| missing measurement | a table omits one of the standard measurements and states no reason |
| off-page name | a page sentence names an arm the page does not print (`OFF_PAGE_ARMS`) |
| undisclosed zero ages | a graph table is built from LDBC rows without the fixed age parse and does not carry the sentence saying their ages loaded as zero (BUGS F146, DECISIONS #119) |
| missing LDBC wording | a table built from LDBC data (`export_web.LDBC_DERIVED_TABLES`, or a dataset or size label that names LDBC) does not open with the sentence "These are not LDBC Benchmark Results." and a sentence that starts "How this differs from the LDBC standard:" (`page_check._check_ldbc_notice`) |
| undisclosed override | a table shows an arm that runs a PROTOCOL.md section 7 override (`override_registry.py`) and prints no sentence saying so, or a row of such an arm lacks the engine's own read-back of the setting or carries another value (`fairness_check` F15; CAMPAIGN.md section 7, item 21) |

## What the page may say while a campaign runs (DECISIONS #102)
Expand All @@ -285,6 +286,8 @@ Once, at the end, linked from every table: the global conditions; the environmen

Beside every peak-memory column with a JVM engine on it, not only the ones that compare ArcadeDB variants: an arm that pins `-Xms=-Xmx` commits the heap, so its column measures reservation, not demand. The ArcadeDB arms, embedded and served, leave `-Xms` unset, so their column measures demand up to `-Xmx`, and the sentence names only the arms whose rows record a pinned heap. The sentence is generated (`export_web._jvm_memory_note`) from the rows. This never clears; it is stated, not fixed.

**LDBC wording (fair use).** [LDBC's fair-use policy](https://ldbcouncil.org/benchmarks/fair-use-policies) lets only a result audited by an LDBC-approved auditor be called an LDBC Benchmark Result; unaudited or derived work must say it is not one, credit LDBC under CC BY 4.0, and say how it differs from the standard. Every table built from LDBC data (`l2` graph OLTP, `l2olap` graph OLAP, and `restart` for its graph rows) therefore opens with `export_web.LDBC_NOTICE` ("These are not LDBC Benchmark Results", the credit, no audit, no endorsement) and its own deviations sentence (`OCT_PROSE[table]["ldbc_deviations"]`: which data and scale factors, which queries are ours and which are LSQB's, no LDBC driver or ACID/recovery test or price/performance, our own harness, embedded and containerised engines). Both are registered sentences, not prose typed per page, and `page_check._check_ldbc_notice` fails a publish when a table lacks either. A new table that reads LDBC data is added to `LDBC_DERIVED_TABLES` in the same change. No number from these tables may be called an LDBC result, or compared with an audited LDBC result, anywhere (page, paper, issue, post).

And a closing list of what the page does NOT measure: concurrency and load (it needs a non-Python load generator; our own harness is censored above ~4 clients by GIL queueing, and a bad concurrency table is worse than a disclosed absence); import, export, and backup; replication beyond the failover trial; strict durability for bulk ingest and reads (the timed writes run at both classes and the durability table prints them, DECISIONS #90); updates against a live vector index, where the dense table times an insert and a delete into a built index at the 1M tier and nothing above it (DECISIONS #82d); dense dimensionality above 128; anything across a real network; k other than 10.

---
Expand Down
42 changes: 38 additions & 4 deletions benchmarks/experiments/export_web.py
Original file line number Diff line number Diff line change
Expand Up @@ -4022,6 +4022,16 @@ def _milvus_seal_proportion():
"the stopped container. Neo4j's value includes the transaction-log files it preallocates "
"in 256 MiB steps, which is how Neo4j uses disk, so they stay in.")

# LDBC's fair-use policy: the disclaimer wording is LDBC's own ("These are not LDBC Benchmark Results"). The tables
# built from LDBC data are listed in LDBC_DERIVED_TABLES; page_check._check_ldbc_notice holds each of them to
# LDBC_NOTICE and to a sentence that starts LDBC_DEVIATIONS_LEAD.
LDBC_NOTICE = (
"These are not LDBC Benchmark Results. This table is derived from the LDBC Social Network Benchmark and credits the "
"Linked Data Benchmark Council (LDBC) under CC BY 4.0; we ran it ourselves, it was not audited by an "
"LDBC-approved auditor, and LDBC has not endorsed it.")
LDBC_DEVIATIONS_LEAD = "How this differs from the LDBC standard:"
LDBC_DERIVED_TABLES = ("l2", "l2olap", "restart")

# A pin is (regex with one capture, fn(P, rows) -> number) or the same with a
# third element "const" when the number is a lane or runner constant.
OCT_PROSE = {
Expand All @@ -4048,6 +4058,11 @@ def _milvus_seal_proportion():
},
"*": {
"disk": (OCT_DISK_NOTE, []),
# LDBC'S FAIR-USE POLICY (https://ldbcouncil.org/benchmarks/fair-use-policies): only a result audited by an
# LDBC-approved auditor may be called an LDBC Benchmark Result; unaudited or derived work says it is not one,
# credits LDBC under CC BY, and says how it differs from the standard. Every table built from LDBC data
# opens with this sentence and then its own deviations (OCT_TABLE_PROSE; page_check._check_ldbc_notice).
"ldbc": (LDBC_NOTICE, []),
},
# e4's explanatory sentences: neither states a measured quantity, so neither
# carries a pin.
Expand Down Expand Up @@ -4131,10 +4146,24 @@ def _milvus_seal_proportion():
},
"l2": {
"projection": ("Every engine traverses the same persons-and-KNOWS projection, with edges stored in both directions.", []),
"ldbc_deviations": (LDBC_DEVIATIONS_LEAD + " the data is the persons-and-KNOWS projection of LDBC's published Interactive "
"datasets at SF1 and SF10, without the messages, forums and tags; the seven operations are our own "
"templates (point lookup, one hop, two hops, three hops filtered on the far end, insert, update and "
"delete), not the Interactive complex, short or update queries; there is no LDBC driver, no operation "
"mix, no throughput-at-target-latency run, no ACID or recovery test and no price/performance figure; "
"the harness is our own Python code, and the engines are embedded in that process or served from "
"containers under one fixed CPU set and memory cap, not a system under test as an audit describes it.", []),
"ingest": ("Ingest paths: ArcadeDB embedded loads through the Python package's graph_batch, the engine's bulk graph loader, with the WAL on, fed 5,000 records at a time; served streams the vertices and edges as JSONL to the HTTP batch endpoint with the WAL on; Neo4j and Memgraph UNWIND batches over bolt; FalkorDB the same UNWIND batches over the Redis protocol; LadybugDB COPY from CSV, its native bulk path; DuckPGQ registers each batch as an Arrow table and INSERT ... SELECTs from it into the Person and knows tables, then defines the property graph over them; PostgreSQL + AGE writes each vertex label and each edge label, the message half included, with one COPY ... FROM STDIN stream into AGE's own label tables, with the ids AGE's sequences would assign, then builds a btree on the Person id property and runs ANALYZE, all inside the build timer; ArangoDB import_bulk; MongoDB insert_many; SurrealDB inserts the persons through its Python SDK and the KNOWS edges as bulk relation inserts.",
[(r"fed ([\d,]+) records at a time", lambda P, rows: _const("l2_graph", "INGEST_BATCH"), "const")]),
},
"l2olap": {
"ldbc_deviations": (LDBC_DEVIATIONS_LEAD + " the data is the full SF1 network of LDBC's published Interactive datasets, not a "
"Business Intelligence dataset; the nine pattern-matching counts are LSQB's (the Labelled Subgraph "
"Query Benchmark, which LDBC hosts), taken nearly verbatim from its published Cypher, with the "
"friends-of-friends pair excluded at this size, and the other queries are our own; there is no LDBC "
"Business Intelligence workload, driver, update microbatch, power or throughput test, no audit and no "
"price/performance figure; the harness is our own Python code, and the engines are embedded in that "
"process or served from containers under one fixed CPU set and memory cap.", []),
"gav": ("The Graph Analytical View is a copy of the graph that ArcadeDB builds in memory, laid out for questions that sweep the whole graph rather than follow a few links. Rows labelled GAV ran with it built, once, before any query was timed, and the view build column is what that took.", []),
},
"e2atom": {
Expand Down Expand Up @@ -4194,6 +4223,10 @@ def _milvus_seal_proportion():
# The server restart (DECISIONS #139 item 2). Facts about the protocol, no
# numbers, so no pins: the counts (cycles, batch size) are on every row.
OCT_PROSE["restart"] = {
"ldbc_deviations": (LDBC_DEVIATIONS_LEAD + " the graph rows restart an engine holding the persons-and-KNOWS projection of LDBC's "
"published Interactive datasets at SF1 and SF10, and a restart time is not an LDBC measure; no LDBC "
"workload, driver, audit or price/performance figure is involved, and the harness is our own Python code. "
"The documents, vector and time series rows contain no LDBC data.", []),
"what": ("A served engine has no open or close of its own to time, so this table times what it has instead: a "
"restart. Each engine is stopped the way its own image stops it, started again on the same data in the "
"same container, and timed until a fixed read answers exactly as it did before the stop. Start is the "
Expand Down Expand Up @@ -4222,11 +4255,11 @@ def _milvus_seal_proportion():
OCT_TABLE_PROSE = {
"l3s": ["recall", "one_timer", "es_pruning"],
"l3d": ["metric", "degree"],
"l2": ["projection"],
"l2olap": ["gav"],
"l2": ["ldbc", "ldbc_deviations", "projection"],
"l2olap": ["ldbc", "ldbc_deviations", "gav"],
"e2atom": ["trial"],
"e2": ["atomic", "interesting", "disk_split"],
"restart": ["what", "stops", "warm_cache", "durability", "models"],
"restart": ["ldbc", "ldbc_deviations", "what", "stops", "warm_cache", "durability", "models"],
}


Expand All @@ -4240,7 +4273,8 @@ def _oct_entries(table_id):


def _R(table_id, key):
return OCT_PROSE[table_id][key][0]
entry = (OCT_PROSE.get(table_id) or {}).get(key) or OCT_PROSE["*"][key]
return entry[0]


_WORDS = {1: "One", 2: "Two", 3: "Three", 4: "Four", 5: "Five",
Expand Down
31 changes: 30 additions & 1 deletion benchmarks/experiments/page_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -613,6 +613,10 @@ def main() -> int:
print("\nthe dense table's ArangoDB IVF sentence says what the harness does, from the rows")
iv_bad = _check_arango_ivf_sentence(payload, rows)
print(f" {iv_bad} IVF sentence finding(s)")
print("\nevery table built from LDBC data says it is not an LDBC Benchmark Result and how it differs "
"(LDBC fair-use policy)")
l_notice_bad = _check_ldbc_notice(payload)
print(f" {l_notice_bad} table(s) missing the wording")
print("\nevery override a table meets is named under it (CAMPAIGN section 7 row 21)")
ov_bad = _check_override_disclosures(payload, rows)
print(f" {ov_bad} undisclosed override(s)")
Expand All @@ -623,7 +627,7 @@ def main() -> int:
j_bad = _check_jvm_defaults_disclosure(payload)
print(f" {j_bad} table(s) missing the sentence")
return 1 if (i_bad or d_bad or a_bad or l_bad or c_bad
or r_bad or m_bad or not u_ok or k_bad or o_bad or n_bad or iv_bad or ov_bad or j_bad) else 0
or r_bad or m_bad or not u_ok or k_bad or o_bad or n_bad or iv_bad or ov_bad or j_bad or l_notice_bad) else 0


# --------------------------------------------------------------------------
Expand Down Expand Up @@ -1435,6 +1439,7 @@ def _check_lane_roster(payload):
# Literal facts that are neither a row nor a lane constant, each with why.
CONDITION_EXEMPT = [
(r"preallocates in 256 MiB steps", "Neo4j's documented transaction-log rotation size"),
(r"CC BY 4\.0", "the licence LDBC's fair-use policy asks derived work to credit"),
]


Expand Down Expand Up @@ -1527,6 +1532,30 @@ def _check_conditions(payload, rows):
return bad


def _check_ldbc_notice(payload):
"""A table built from LDBC data opens with LDBC's disclaimer and lists its deviations from the standard.

LDBC's fair-use policy (https://ldbcouncil.org/benchmarks/fair-use-policies) lets only an audited result be called an LDBC
Benchmark Result; unaudited or derived work says it is not one, credits LDBC under CC BY, and says how it differs. A table is
LDBC-derived when it is one of export_web.LDBC_DERIVED_TABLES or its dataset or a size label names LDBC. The exporter adds
both sentences from the registry; this holds the payload to the exact text, so a refactor that drops them, or rewords the
disclaimer, fails here and not on the published page. Returns bad count."""
import export_web as EW
bad = 0
for t in payload.get("tables", []):
named = ("LDBC" in str(t.get("dataset") or "")
or any("LDBC" in str(e.get("scale_label") or "") for e in t.get("entries") or []))
if t.get("id") not in EW.LDBC_DERIVED_TABLES and not named:
continue
conds = [str(c) for c in (t.get("conditions") or [])]
if EW.LDBC_NOTICE in conds and any(c.startswith(EW.LDBC_DEVIATIONS_LEAD) for c in conds):
print(f" {t['id']}: carries the LDBC wording")
else:
print(f" MISSING {t['id']}: built from LDBC data and does not carry LDBC's disclaimer and its deviations")
bad += 1
return bad


def _check_override_disclosures(payload, rows=None):
"""A table that shows an arm running a PROTOCOL section 7 override prints a
sentence that says so (override_registry.py, CAMPAIGN section 7 row 21).
Expand Down
40 changes: 40 additions & 0 deletions benchmarks/experiments/test_page_check_2611.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,3 +148,43 @@ def test_the_gate_run_refuses_a_payload_of_another_instrument(built, tmp_path):
env=E2E._env(results), capture_output=True, text=True, timeout=900)
assert done.returncode == 1 and "WRONG INSTRUMENT" in done.stdout
assert "Traceback" not in done.stderr


def test_every_ldbc_table_says_it_is_not_an_ldbc_benchmark_result(at, capsys):
_results, payload = at
ldbc = [t for t in payload["tables"] if t["id"] in ("l2", "l2olap", "restart")]
assert ldbc, "the fixture payload carries no LDBC-derived table"
for t in ldbc:
assert "These are not LDBC Benchmark Results." in " ".join(t["conditions"])
assert PC._check_ldbc_notice(payload) == 0


def test_a_table_that_lost_the_ldbc_wording_is_caught(at, capsys):
_results, payload = at
table = next(t for t in payload["tables"] if t["id"] == "l2")
table["conditions"] = [c for c in table["conditions"] if "LDBC" not in c]
assert PC._check_ldbc_notice(payload) == 1
assert "MISSING l2" in capsys.readouterr().out


def test_a_reworded_ldbc_disclaimer_is_caught(at):
_results, payload = at
table = next(t for t in payload["tables"] if t["id"] == "l2")
table["conditions"] = [c.replace("These are not LDBC Benchmark Results.", "Inspired by LDBC.") for c in table["conditions"]]
assert PC._check_ldbc_notice(payload) == 1


def test_the_ldbc_sentences_carry_no_unpinned_number(at, capsys):
"""The gate that reads every condition sentence for numbers sees none it cannot place in the LDBC wording."""
import export_web as EW
_results, payload = at
PC._check_conditions(payload, [])
out = capsys.readouterr().out
assert not [l for l in out.splitlines() if ("UNPINNED" in l or "UNREGISTERED" in l) and "LDBC" in l], out
spans = lambda text: [m.group(0) for m in PC.CONDITION_TOKEN.finditer(text)] # noqa: E731
for tid in EW.LDBC_DERIVED_TABLES:
for key in ("ldbc", "ldbc_deviations"):
text = EW._R(tid, key)
allowed = PC._spans([p for p, _ in PC.CONDITION_ALLOWED] + [p for p, _ in PC.CONDITION_EXEMPT], text)
loose = [t for t in PC.CONDITION_TOKEN.finditer(text) if not PC._covered(t.span(), allowed)]
assert not loose, (tid, key, [t.group(0) for t in loose])
Loading