Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions server/src/mcp-handler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import type { PonderClient } from "./ponder-client.js"
import type { ContractReader } from "./contract-reader.js"
import { DealbotClient } from "./dealbot-client.js"
import { formatTableList } from "./table-metadata.js"
import { sqlNotices } from "./tripwires.js"
import type { BetterStackClient } from "./betterstack-client.js"
import type { SubgraphClient } from "./subgraph-client.js"
import { ProvingClient } from "./proving-client.js"
Expand Down Expand Up @@ -120,14 +121,15 @@ Tables and key columns:
${formatTableList()}

Key joins: fwss tables use data_set_id, pdp tables use set_id, these are the same value (JOIN fwss.data_set_id = pdp.set_id). rail_id across fp tables. Link rails to datasets: JOIN fwss_data_set_created.pdp_rail_id = fp_rail_settled.rail_id. Provider names: call get_providers first, then JOIN by provider_id.
Amounts are bigint (18 decimals). Gas cost in FIL = m.gas_used * m.effective_gas_price / 1e18 (m being tx_meta).`,
Amounts are bigint but decimals are PER-TOKEN: USDFC/FIL = 18 (1e18), axlUSDC = 6 (1e6). JOIN fp_rail_created.token and scale per token; never divide a mixed-token SUM by a fixed 1e18. Gas in FIL = m.gas_used * m.effective_gas_price / 1e18 (m = tx_meta).`,
inputSchema: { i_have_read_the_system_context: affirmation, network: networkEnum, sql: z.string().describe("SQL SELECT query to execute") },
}, logged("query_sql", async ({ network, sql }) => {
const sqlStart = Date.now()
try {
const result = await getPonder(network).querySql(sql)
logSql("mcp", network, sql, Date.now() - sqlStart, { rowCount: result.rowCount })
return toolResult({ network, ...result })
const notices = sqlNotices(sql, result)
return toolResult({ network, ...result, ...(notices.length ? { notices } : {}) })
} catch (err) {
logSql("mcp", network, sql, Date.now() - sqlStart, { error: sanitizeError(err) })
return toolError(err)
Expand Down
4 changes: 3 additions & 1 deletion server/src/routes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import { SubgraphClient } from "./subgraph-client.js"
import { ProvingClient } from "./proving-client.js"
import type { NetworkName } from "./networks.js"
import { logRest, logSql } from "./logger.js"
import { sqlNotices } from "./tripwires.js"

interface Backends {
ponderClients: Map<NetworkName, PonderClient>
Expand Down Expand Up @@ -84,7 +85,8 @@ export function createRoutes(backends: Backends): Hono {
try {
const result = await ctx.ponder.querySql(body.sql)
logSql("rest", ctx.network, body.sql, Date.now() - sqlStart, { rowCount: result.rowCount })
return c.json({ network: ctx.network, ...result })
const notices = sqlNotices(body.sql, result)
return c.json({ network: ctx.network, ...result, ...(notices.length ? { notices } : {}) })
} catch (err) {
logSql("rest", ctx.network, body.sql, Date.now() - sqlStart, { error: sanitizeError(err) })
return c.json({ network: ctx.network, error: sanitizeError(err) }, 400)
Expand Down
55 changes: 55 additions & 0 deletions server/src/tripwires.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
/**
* Result-side tripwires for query_sql.
*
* Cheap heuristics over the SQL text + results that warn about observed
* recurring correctness traps.
*/
import type { SqlResult } from "./ponder-client.js"

const AXLUSDC = "0xeb466342c4d449bc9f53a865d5cb90586f405215"

export function sqlNotices(sql: string, result: SqlResult): string[] {
const notices: string[] = []
const q = sql.toLowerCase()

// -- Two payment channels: fp_rail_settled is streaming only --
const settled = q.includes("fp_rail_settled")
const oneTime = q.includes("fp_one_time_payment")
const aggregating = /\bsum\s*\(/.test(q) || /\brevenue\b/.test(q) || /\bvolume\b/.test(q)
if (settled && !oneTime && aggregating) {
notices.push(
"fp_rail_settled is only the STREAMING channel. fp_one_time_payment is a parallel channel (often larger) with no gross column (gross = net_payee_amount + network_fee + operator_commission). Total volume = both; do not total from fp_rail_settled alone.",
)
}

// -- Per-token decimals: a fixed /1e18 zeroes 6-decimal axlUSDC --
// Only FilecoinPay amounts carry a token dimension. tx_meta gas is always FIL
// and /1e18 is its correct scaling, so gas-only queries must stay quiet.
const touchesPayments = /\bfp_[a-z_]+/.test(q)
const dividesBy1e18 = /\/\s*1e18\b/.test(q) || /\/\s*1_?0{18}\b/.test(q)
const sumsAmount =
/\bsum\s*\(/.test(q) && /(amount|net_payee|network_fee|operator_commission|value|price)/.test(q)
const mentionsToken = /\btoken\b/.test(q)
if (touchesPayments && dividesBy1e18) {
notices.push(
"Decimals are per-token: axlUSDC = 6 (1e6), USDFC/FIL = 18 (1e18). A fixed /1e18 collapses axlUSDC value to ~0. JOIN fp_rail_created.token and scale each token by its own decimals.",
)
} else if (touchesPayments && sumsAmount && !mentionsToken) {
notices.push(
"Summing amounts without GROUP BY token mixes 6-decimal (axlUSDC) and 18-decimal (USDFC/FIL) values. Scale per token before summing.",
)
}

// -- Result-shape: a result set spanning multiple tokens including axlUSDC --
const tokenCol = result.columns.find((c) => c.toLowerCase() === "token")
if (tokenCol && result.rows.length > 1) {
const tokens = new Set(result.rows.map((r) => String(r[tokenCol]).toLowerCase()))
if (tokens.size > 1 && tokens.has(AXLUSDC)) {
notices.push(
"This result spans multiple tokens including axlUSDC (6-decimal). Scale each row by its own token's decimals; do not divide the whole set by one scalar.",
)
}
}

return notices
}
83 changes: 83 additions & 0 deletions server/test/tripwires.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
import { describe, expect, test } from "vitest"
import { sqlNotices } from "../src/tripwires.js"

const AXLUSDC = "0xeb466342c4d449bc9f53a865d5cb90586f405215"
const USDFC = "0x80b98d3aa09ffff255c3ba4a241111ff1262f045"
const empty = { columns: [], rows: [], rowCount: 0 }

const has = (notices: string[], substr: string) => notices.some((n) => n.includes(substr))

describe("sqlNotices tripwires", () => {
test("two-channel: fp_rail_settled SUM without fp_one_time_payment warns", () => {
const n = sqlNotices("SELECT SUM(total_net_payee_amount) FROM fp_rail_settled", empty)
expect(has(n, "fp_one_time_payment")).toBe(true)
})

test("two-channel: silent when both channels present", () => {
const n = sqlNotices(
"SELECT SUM(s.total_net_payee_amount) FROM fp_rail_settled s UNION ALL SELECT SUM(o.net_payee_amount) FROM fp_one_time_payment o",
empty,
)
expect(has(n, "fp_one_time_payment is a parallel")).toBe(false)
})

test("per-token: a fixed /1e18 warns", () => {
const n = sqlNotices("SELECT SUM(total_settled_amount)/1e18 FROM fp_rail_settled", empty)
expect(has(n, "collapses axlUSDC")).toBe(true)
})

Comment thread
rvagg marked this conversation as resolved.
Resolved
test("per-token: SUM(amount) without token grouping warns", () => {
const n = sqlNotices("SELECT SUM(amount) FROM fp_deposit", empty)
expect(has(n, "without GROUP BY token")).toBe(true)
})

test("per-token: SUM with token grouping does not warn about mixing", () => {
const n = sqlNotices("SELECT token, SUM(amount) FROM fp_deposit GROUP BY token", empty)
expect(has(n, "without GROUP BY token")).toBe(false)
})

test("result-shape: mixed-token result including axlUSDC warns", () => {
const result = {
columns: ["token", "n"],
rows: [{ token: USDFC, n: "5" }, { token: AXLUSDC, n: "3" }],
rowCount: 2,
}
const n = sqlNotices("SELECT token, COUNT(*) n FROM fp_rail_created GROUP BY token", result)
expect(has(n, "spans multiple tokens")).toBe(true)
})

test("result-shape: single-token result does not warn", () => {
const result = { columns: ["token", "n"], rows: [{ token: USDFC, n: "5" }], rowCount: 1 }
const n = sqlNotices("SELECT token, COUNT(*) n FROM fp_rail_created GROUP BY token", result)
expect(has(n, "spans multiple tokens")).toBe(false)
})

test("per-token: the documented tx_meta gas recipe stays quiet", () => {
const n = sqlNotices(
"SELECT tx_hash, gas_used * effective_gas_price / 1e18 AS fil FROM tx_meta ORDER BY fil DESC LIMIT 10",
empty,
)
expect(n).toHaveLength(0)
})

test("per-token: SUM over gas without payments data stays quiet", () => {
const n = sqlNotices(
"SELECT SUM(gas_used * effective_gas_price) FROM tx_meta WHERE block_number > 6000000",
empty,
)
expect(n).toHaveLength(0)
})

test("per-token: gas scaling alongside payments data still warns", () => {
const n = sqlNotices(
"SELECT SUM(s.total_settled_amount)/1e18 FROM fp_rail_settled s JOIN tx_meta m USING (tx_hash)",
empty,
)
expect(has(n, "collapses axlUSDC")).toBe(true)
})

test("clean analytical query produces no notices", () => {
const n = sqlNotices("SELECT provider_id, COUNT(*) FROM fwss_data_set_created GROUP BY provider_id", empty)
expect(n).toHaveLength(0)
})
})
16 changes: 8 additions & 8 deletions shared/system-context.ts
Original file line number Diff line number Diff line change
Expand Up @@ -543,14 +543,14 @@ Three places to get fault/proving data:

## Aggregate FilecoinPay Metrics (network-wide, all operators)

CRITICAL: For network-wide metrics, start from fp_* tables. Do NOT join to fwss_* tables, since that only captures FWSS-operated rails and misses other operators like Storacha (which accounts for ~74% of mainnet settlement volume).
CRITICAL: For network-wide metrics, start from fp_* tables. Do NOT join to fwss_* tables, which only capture FWSS-operated rails and miss every other operator on FilecoinPay (Storacha [wound down], PoRep Market, and occasional one-offs). Operator shares shift; compute them per question (Revenue by operator).

**Total revenue**: SUM(total_net_payee_amount::numeric)/1e18 FROM fp_rail_settled. This is all USDFC paid to all SPs across all operators.
**Total revenue including one-time payments**: Add SUM(net_payee_amount::numeric)/1e18 FROM fp_one_time_payment.
**ARR (Annual Recurring Revenue)**: Use fp_rail_rate_modified (NOT fwss_rail_rate_updated which is FWSS-only). For active non-terminated non-finalized rails: SUM the latest new_rate per rail_id, multiply by epochs_per_year (2880 * 365). Or query live rail state via get_rail for each active rail.
**Revenue by operator**: JOIN fp_rail_settled to fp_rail_created on rail_id, GROUP BY operator. This shows FWSS vs Storacha vs other operators.
**Revenue by SP**: JOIN fp_rail_settled to fp_rail_created on rail_id, GROUP BY payee. Each payee is an SP address.
**Deposits/TVL**: fp_deposit and fp_withdrawal directly, no FWSS join needed.
**Total revenue (per token)**: JOIN fp_rail_settled to fp_rail_created on rail_id, GROUP BY rc.token; divide each token's SUM(total_net_payee_amount::numeric) by its decimals (USDFC/FIL 1e18, axlUSDC 1e6).
**Two payment channels - total volume is BOTH**: also SUM fp_one_time_payment (no gross column; gross = net_payee_amount+network_fee+operator_commission), scaled per-token. One-time often exceeds streaming, so never quote total volume from fp_rail_settled alone.
**ARR (Annual Recurring Revenue)**: Use fp_rail_rate_modified (NOT fwss_rail_rate_updated which is FWSS-only). For active non-terminated non-finalized rails: SUM the latest new_rate per rail_id, multiply by epochs_per_year (2880 * 365), per token. Or query live rail state via get_rail.
**Revenue by operator**: JOIN fp_rail_settled to fp_rail_created on rail_id, GROUP BY operator, token.
**Revenue by SP**: same JOIN, GROUP BY payee, token. Each payee is an SP address.
**Deposits/TVL**: fp_deposit and fp_withdrawal directly, GROUP BY token, no FWSS join needed.

Known operators on mainnet:
- FWSS: 0x8408502033c418e1bbc97ce9ac48e5528f371a9f
Expand Down Expand Up @@ -649,7 +649,7 @@ For gas analysis: JOIN your event table to tx_meta USING (tx_hash). Piece count

**Total data stored**: SUM(raw_size) from fwss_piece_added gives total bytes of original (unpadded) data. Divide by 1e12 for TiB. Filter by provider via JOIN with fwss_data_set_created. Exclude terminated datasets by LEFT JOIN with fwss_service_terminated and filtering WHERE terminated IS NULL.

**Total revenue**: SUM(total_net_payee_amount) / 1e18 from fp_rail_settled gives all-time USDFC paid to SPs. Join with fwss_data_set_created via pdp_rail_id for per-provider breakdown. Remember: these fields are INCREMENTAL per event, so SUM() is correct.
**Total revenue**: use the two-channel, per-token recipe in Aggregate FilecoinPay Metrics (fp_rail_settled + fp_one_time_payment; scale by token decimals). Join fwss_data_set_created via pdp_rail_id for per-FWSS-provider breakdown. Fields are INCREMENTAL per event, so SUM() is correct.

**Time filtering**: Use timestamp column (unix seconds). Last 7 days: WHERE timestamp > EXTRACT(EPOCH FROM NOW()) - 7*86400. By date: WHERE TO_TIMESTAMP(timestamp) >= '2026-03-01'. By epoch: WHERE block_number > 5860000.

Expand Down