Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .claude/skills/contribute-benchmark/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Walks a contributor through adding a new benchmark to OpenChainBenc

You are helping someone add a new benchmark to OpenChainBench, an open registry of crypto-infrastructure benchmarks.

If the user does not already have the repo cloned, your first step is to suggest `git clone https://github.com/ChainBench/OpenChainBench && cd OpenChainBench`. Everything below assumes that working directory.
If the user does not already have the repo cloned, your first step is to suggest `git clone https://github.com/OpenChainBench/OpenChainBench && cd OpenChainBench`. Everything below assumes that working directory.

## Mental model

Expand All @@ -24,7 +24,7 @@ Walk the user through these six steps in order. Do not skip ahead.

### 1. Open an issue

Direct them to the `Propose a benchmark` issue template at https://github.com/ChainBench/OpenChainBench/issues/new?template=new-benchmark.yml. Maintainers need to see the proposed metric, providers, methodology and harness location before code review. Brainstorm-stage ideas belong in https://github.com/ChainBench/OpenChainBench/discussions/categories/ideas instead.
Direct them to the `Propose a benchmark` issue template at https://github.com/OpenChainBench/OpenChainBench/issues/new?template=new-benchmark.yml. Maintainers need to see the proposed metric, providers, methodology and harness location before code review. Brainstorm-stage ideas belong in https://github.com/OpenChainBench/OpenChainBench/discussions/categories/ideas instead.

### 2. Write the spec

Expand Down
4 changes: 2 additions & 2 deletions .github/CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ OpenChainBench publishes performance numbers about real services run by real peo
## Expected behavior

- **Critique numbers, not people.** It's fair to question a methodology, a sample window, a regional bias. It's not fair to suggest someone is incompetent or dishonest.
- **Cite, don't assert.** If you think a published figure is wrong, file a [provider correction](https://github.com/ChainBench/OpenChainBench/issues/new?template=provider-correction.yml) with a reproducer. "This number is bullshit" without a reproducer is a non-issue.
- **Cite, don't assert.** If you think a published figure is wrong, file a [provider correction](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=provider-correction.yml) with a reproducer. "This number is bullshit" without a reproducer is a non-issue.
- **Disclose conflicts.** If you work for a provider that's measured here, say so when you participate in discussions about that provider. It doesn't disqualify your input. it just keeps the conversation honest.
- **Respect contributors' time.** Maintainers are not obligated to debate. If a thread is going in circles, a maintainer may close it with a dated decision and that decision stands until new evidence is filed.

Expand All @@ -28,7 +28,7 @@ Decisions are recorded in the affected thread.

## Reporting

For private reports. including suspected manipulation, harassment, or undisclosed conflicts of interest. email **conduct@openchainbench.com** or open a [private security advisory](https://github.com/ChainBench/OpenChainBench/security/advisories/new) (the same channel works for non-security disclosures).
For private reports. including suspected manipulation, harassment, or undisclosed conflicts of interest. email **conduct@openchainbench.com** or open a [private security advisory](https://github.com/OpenChainBench/OpenChainBench/security/advisories/new) (the same channel works for non-security disclosures).

We treat reporters' identities as confidential by default.

Expand Down
4 changes: 2 additions & 2 deletions .github/DISCUSSIONS_AND_PROJECT_SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Default categories like "General" and "Polls" should be deleted to keep the surf

- **Name**: `OpenChainBench Roadmap`
- **Visibility**: Public
- **Linked repository**: `ChainBench/OpenChainBench`
- **Linked repository**: `OpenChainBench/OpenChainBench`

### Columns

Expand All @@ -41,7 +41,7 @@ Requested → Approved → In progress → Live → Archived

In Project Settings → Workflows, enable:

- **Auto-add to project**. for issues in `ChainBench/OpenChainBench` matching the filter `is:issue label:bench-request`. They land in `Requested` automatically.
- **Auto-add to project**. for issues in `OpenChainBench/OpenChainBench` matching the filter `is:issue label:bench-request`. They land in `Requested` automatically.
- **Item closed → Archived**. when an issue is closed without a `live` label, move it to `Archived`.
- **PR merged → Live** (manual; GitHub doesn't have a native trigger, so a maintainer moves the card after merge).

Expand Down
8 changes: 4 additions & 4 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
blank_issues_enabled: false
contact_links:
- name: 💡 Idea or rough proposal
url: https://github.com/ChainBench/OpenChainBench/discussions/new?category=ideas
url: https://github.com/OpenChainBench/OpenChainBench/discussions/new?category=ideas
about: Have an idea but not ready for a formal benchmark proposal? Start a discussion in Ideas.
- name: 🙋 Question (How do I…?)
url: https://github.com/ChainBench/OpenChainBench/discussions/new?category=q-a
url: https://github.com/OpenChainBench/OpenChainBench/discussions/new?category=q-a
about: Ask a question about a spec, harness, methodology, or how the site works.
- name: 📊 Show & tell
url: https://github.com/ChainBench/OpenChainBench/discussions/new?category=show-and-tell
url: https://github.com/OpenChainBench/OpenChainBench/discussions/new?category=show-and-tell
about: Show a fork, a dashboard, or a benchmark you ran locally.
- name: 🔐 Security disclosure
url: https://github.com/ChainBench/OpenChainBench/security/advisories/new
url: https://github.com/OpenChainBench/OpenChainBench/security/advisories/new
about: Privately report a vulnerability. Do not file a public issue for security problems.
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/new-benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ body:

OpenChainBench prefers to align on **what** is measured and **how** before any code is written. Fill out the sections below. fields marked required are the bare minimum needed for the maintainers to give meaningful feedback. If you're not sure on some of them, write what you have and leave a note.

Have a rough idea but not ready to fill all this? Open a thread in [Discussions → Ideas](https://github.com/ChainBench/OpenChainBench/discussions/categories/ideas) instead.
Have a rough idea but not ready to fill all this? Open a thread in [Discussions → Ideas](https://github.com/OpenChainBench/OpenChainBench/discussions/categories/ideas) instead.
- type: input
id: metric
attributes:
Expand Down Expand Up @@ -97,5 +97,5 @@ body:
options:
- label: I understand that the harness code, methodology, and any corrections will be published under the project's MIT/CC-BY-4.0 license.
required: true
- label: I have searched [existing benchmarks](https://github.com/ChainBench/OpenChainBench/tree/main/benchmarks) and [open issues](https://github.com/ChainBench/OpenChainBench/issues?q=is%3Aissue+label%3Abench-request) for duplicates.
- label: I have searched [existing benchmarks](https://github.com/OpenChainBench/OpenChainBench/tree/main/benchmarks) and [open issues](https://github.com/OpenChainBench/OpenChainBench/issues?q=is%3Aissue+label%3Abench-request) for duplicates.
required: true
2 changes: 1 addition & 1 deletion .github/SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Reporting a vulnerability

Open a [private security advisory](https://github.com/ChainBench/OpenChainBench/security/advisories/new) on this repository. Do **not** open a public issue for a security report. the advisory mechanism keeps the disclosure private until a fix is in place.
Open a [private security advisory](https://github.com/OpenChainBench/OpenChainBench/security/advisories/new) on this repository. Do **not** open a public issue for a security report. the advisory mechanism keeps the disclosure private until a fix is in place.

If GitHub is not an option, email **security@openchainbench.com** with:

Expand Down
14 changes: 7 additions & 7 deletions .github/SUPPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@ OpenChainBench is community-run. Before opening an issue, pick the right channel

| Goal | Where to go |
| --- | --- |
| Propose a new benchmark formally | [New issue → 📊 Propose a benchmark](https://github.com/ChainBench/OpenChainBench/issues/new?template=new-benchmark.yml) |
| Float an idea before writing it up | [Discussions → Ideas](https://github.com/ChainBench/OpenChainBench/discussions/categories/ideas) |
| Ask a question about a spec, harness, methodology, or the site | [Discussions → Q&A](https://github.com/ChainBench/OpenChainBench/discussions/categories/q-a) |
| Show a fork or a dashboard you built | [Discussions → Show & tell](https://github.com/ChainBench/OpenChainBench/discussions/categories/show-and-tell) |
| Report a number that looks wrong on the site | [New issue → 🐞 Data quality](https://github.com/ChainBench/OpenChainBench/issues/new?template=data-quality.yml) |
| Submit a provider correction (you measured a different number) | [New issue → ✏️ Provider correction](https://github.com/ChainBench/OpenChainBench/issues/new?template=provider-correction.yml) |
| Privately report a security vulnerability | [Security advisories](https://github.com/ChainBench/OpenChainBench/security/advisories/new) |
| Propose a new benchmark formally | [New issue → 📊 Propose a benchmark](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=new-benchmark.yml) |
| Float an idea before writing it up | [Discussions → Ideas](https://github.com/OpenChainBench/OpenChainBench/discussions/categories/ideas) |
| Ask a question about a spec, harness, methodology, or the site | [Discussions → Q&A](https://github.com/OpenChainBench/OpenChainBench/discussions/categories/q-a) |
| Show a fork or a dashboard you built | [Discussions → Show & tell](https://github.com/OpenChainBench/OpenChainBench/discussions/categories/show-and-tell) |
| Report a number that looks wrong on the site | [New issue → 🐞 Data quality](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=data-quality.yml) |
| Submit a provider correction (you measured a different number) | [New issue → ✏️ Provider correction](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=provider-correction.yml) |
| Privately report a security vulnerability | [Security advisories](https://github.com/OpenChainBench/OpenChainBench/security/advisories/new) |
| See what's planned and where to contribute code | [Roadmap project board](https://github.com/orgs/OpenChainBench/projects) |
| Read about how the data is produced | [`benchmarks/README.md`](../benchmarks/README.md) and [`harnesses/README.md`](../harnesses/README.md) |

Expand Down
12 changes: 11 additions & 1 deletion .github/workflows/staging-deploy.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
name: Vercel staging deploy

# Auto-deploy to Vercel Preview on every push to dev. Workaround for the
# Vercel ↔ GitHub App pairing being unreliable across the OpenChainBench
# (user account) + mobula-labs (Vercel team) setup. The Action calls the
# Vercel CLI directly with VERCEL_TOKEN, bypassing the App webhook flow.
#
# Required GitHub secrets (Settings → Secrets and variables → Actions):
# VERCEL_TOKEN — vercel.com/account/tokens, scope = mobula-labs
# VERCEL_ORG_ID — ***SCRUBBED***
# VERCEL_PROJECT_ID — ***SCRUBBED***

on:
push:
branches:
Expand Down Expand Up @@ -51,7 +61,7 @@ jobs:
vercel alias set "${{ steps.deploy.outputs.url }}" \
staging-openchainbench.vercel.app \
--token=${{ secrets.VERCEL_TOKEN }} \
--scope=${{ secrets.VERCEL_ORG_ID }}
--scope=mobula-labs

- name: Comment summary
run: |
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This version has breaking changes. APIs, conventions, and file structure may all
Two long-lived branches:

- `main` → openchainbench.com (production). Vercel auto-deploy on push is **disabled**; production deploys are **manual** via `vercel --prod` from a checkout of `main`.
- `dev` → staging. Each push triggers `.github/workflows/staging-deploy.yml` which calls Vercel CLI with a token and emits a Preview URL.
- `dev` → staging. Each push triggers `.github/workflows/vercel-staging.yml` which calls Vercel CLI with a token and emits a Preview URL. The Vercel GitHub App is NOT used for staging because the OpenChainBench user account + mobula-labs Vercel team pairing wouldn't propagate.

**Standard flow:**

Expand Down
14 changes: 7 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,16 @@ OpenChainBench is a federation. Each harness is hosted by whoever wrote it. the

| Goal | Right channel |
| --- | --- |
| Float a rough idea | [Discussions → Ideas](https://github.com/ChainBench/OpenChainBench/discussions/categories/ideas) |
| Ask a methodology / harness question | [Discussions → Q&A](https://github.com/ChainBench/OpenChainBench/discussions/categories/q-a) |
| Propose a benchmark formally | [Issue → 📊 Propose a benchmark](https://github.com/ChainBench/OpenChainBench/issues/new?template=new-benchmark.yml) |
| Report a number that looks wrong | [Issue → 🐞 Data quality](https://github.com/ChainBench/OpenChainBench/issues/new?template=data-quality.yml) |
| Submit a provider correction | [Issue → ✏️ Provider correction](https://github.com/ChainBench/OpenChainBench/issues/new?template=provider-correction.yml) |
| Float a rough idea | [Discussions → Ideas](https://github.com/OpenChainBench/OpenChainBench/discussions/categories/ideas) |
| Ask a methodology / harness question | [Discussions → Q&A](https://github.com/OpenChainBench/OpenChainBench/discussions/categories/q-a) |
| Propose a benchmark formally | [Issue → 📊 Propose a benchmark](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=new-benchmark.yml) |
| Report a number that looks wrong | [Issue → 🐞 Data quality](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=data-quality.yml) |
| Submit a provider correction | [Issue → ✏️ Provider correction](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=provider-correction.yml) |
| See what's planned | [Roadmap project board](https://github.com/orgs/OpenChainBench/projects) |

## Submitting a benchmark

1. **Open an issue** with the [📊 Propose a benchmark template](https://github.com/ChainBench/OpenChainBench/issues/new?template=new-benchmark.yml). Sketch the metric, providers, methodology, where you'll host the harness. get feedback before you write code. The issue lands in the `Requested` column of the roadmap.
1. **Open an issue** with the [📊 Propose a benchmark template](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=new-benchmark.yml). Sketch the metric, providers, methodology, where you'll host the harness. get feedback before you write code. The issue lands in the `Requested` column of the roadmap.
2. **Write the spec.** Drop a YAML at `benchmarks/<slug>.yml`. The format is described in [`benchmarks/README.md`](./benchmarks/README.md) and validated by `src/lib/spec-schema.ts`.
3. **Build the harness** at `harnesses/<slug>/`. A harness is a data producer only. it exposes `/metrics` over HTTPS with the metric names and labels your spec references. See [`harnesses/README.md`](./harnesses/README.md) for the contract and the existing Go harnesses as reference implementations.
4. **Deploy the harness** on whatever infra fits. Railway, Fly, Cloud Run, a VPS. Expose `/metrics` over HTTPS at a stable public URL. You own the runtime, the secrets and the budget. The maintainers never see your API keys.
Expand All @@ -56,7 +56,7 @@ pnpm build # production build

## Corrections

If you can't reproduce a number, file a [provider correction](https://github.com/ChainBench/OpenChainBench/issues/new?template=provider-correction.yml) (you measured a different value for your service) or a [data quality issue](https://github.com/ChainBench/OpenChainBench/issues/new?template=data-quality.yml) (the site is showing something obviously wrong or stale). Material errors are corrected in place with a dated note on the report.
If you can't reproduce a number, file a [provider correction](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=provider-correction.yml) (you measured a different value for your service) or a [data quality issue](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=data-quality.yml) (the site is showing something obviously wrong or stale). Material errors are corrected in place with a dated note on the report.

## Code of conduct & security

Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ To render the site against your local harness, run a local Prometheus scraping `

Full guide in [CONTRIBUTING.md](./CONTRIBUTING.md). For a concrete end-to-end example, read [`docs/walkthrough.md`](./docs/walkthrough.md). Short version:

1. **Open an issue** with the [📊 Propose a benchmark template](https://github.com/ChainBench/OpenChainBench/issues/new?template=new-benchmark.yml). Sketch the metric, providers, methodology. get feedback before you build. Want to brainstorm first? Use [Discussions → Ideas](https://github.com/ChainBench/OpenChainBench/discussions/categories/ideas) instead.
1. **Open an issue** with the [📊 Propose a benchmark template](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=new-benchmark.yml). Sketch the metric, providers, methodology. get feedback before you build. Want to brainstorm first? Use [Discussions → Ideas](https://github.com/OpenChainBench/OpenChainBench/discussions/categories/ideas) instead.
2. **Write the spec** at `benchmarks/<slug>.yml`. Format documented in [`benchmarks/README.md`](./benchmarks/README.md), validated by `src/lib/spec-schema.ts`.
3. **Build the harness** in `harnesses/<slug>/`. Any language works as long as it exposes `/metrics` over HTTPS with the metric names and labels your spec references. The harness is a data producer only. no Prometheus, Grafana, or Alertmanager packaging.
4. **Deploy the harness** on whatever infra fits. Railway, Fly, Cloud Run, a VPS, even a home server with a static IP. Expose `/metrics` over HTTPS at a stable public URL. You own the runtime, the secrets and the budget.
Expand Down Expand Up @@ -314,11 +314,11 @@ PNG URL — the downloaded image matches what the user is looking at.

## Community

- 💡 [Discussions → Ideas](https://github.com/ChainBench/OpenChainBench/discussions/categories/ideas). brainstorm new benchmarks before writing them up
- 🙋 [Discussions → Q&A](https://github.com/ChainBench/OpenChainBench/discussions/categories/q-a). methodology / harness / spec questions
- 📊 [Discussions → Show & tell](https://github.com/ChainBench/OpenChainBench/discussions/categories/show-and-tell). share forks and dashboards
- 💡 [Discussions → Ideas](https://github.com/OpenChainBench/OpenChainBench/discussions/categories/ideas). brainstorm new benchmarks before writing them up
- 🙋 [Discussions → Q&A](https://github.com/OpenChainBench/OpenChainBench/discussions/categories/q-a). methodology / harness / spec questions
- 📊 [Discussions → Show & tell](https://github.com/OpenChainBench/OpenChainBench/discussions/categories/show-and-tell). share forks and dashboards
- 🗺️ [Roadmap](https://github.com/orgs/OpenChainBench/projects). what's planned and what's live
- 🐞 [New issue](https://github.com/ChainBench/OpenChainBench/issues/new/choose). formal benchmark proposal, data-quality flag, or provider correction
- 🐞 [New issue](https://github.com/OpenChainBench/OpenChainBench/issues/new/choose). formal benchmark proposal, data-quality flag, or provider correction
- See [SUPPORT.md](./.github/SUPPORT.md) for the full triage matrix.

## SEO TODOs
Expand All @@ -339,7 +339,7 @@ Tracking the work to make benchmark pages rank for the queries they target.
- Site. [openchainbench.com](https://openchainbench.com)
- Live stream. folded into [openchainbench.com](https://openchainbench.com)
- Twitter. [@openchainbench](https://twitter.com/openchainbench)
- GitHub. [ChainBench/OpenChainBench](https://github.com/ChainBench/OpenChainBench)
- GitHub. [OpenChainBench/OpenChainBench](https://github.com/OpenChainBench/OpenChainBench)

## License

Expand Down
Loading
Loading