run --local: --trace-otlp to export traces to an OTLP collector

.claude/skills/mendix/README.md

@@ -66,6 +66,7 @@ Page-specific patterns:
 | [create-custom-widget.md](create-custom-widget.md) | Custom pluggable widget AIGC | Creating custom React widgets from scratch |
 | [migrate-design-prototype.md](migrate-design-prototype.md) | Turn a Claude Design prototype into a themed Mendix app | Reproducing a design handoff/prototype as an SCSS theme + styled pages |
 | [debug-bson.md](debug-bson.md) | BSON debugging | Troubleshooting SDK issues |
+| [analyze-runtime.md](analyze-runtime.md) | Analyze runtime behavior — logs, metrics, traces, catalog, and cross-source joins | Profiling a slow page/microflow, finding what hits the DB, correlating cost with model shape |
 
 ---
 
@@ -97,6 +98,8 @@ Load skills based on the task:
 | "Build a pluggable widget" | `create-custom-widget.md` |
 | "Turn a design prototype/handoff into a Mendix app" | `migrate-design-prototype.md`, `theme-styling.md`, `create-page.md` |
 | "Build/apply a theme from a design" | `migrate-design-prototype.md`, `theme-styling.md` |
+| "Why is this slow / profile the app / what hits the database" | `analyze-runtime.md`, `run-local.md` |
+| "Trace / metrics / flame chart / correlate cost with model" | `analyze-runtime.md` |
 
 ### For Error Recovery
 

.claude/skills/mendix/analyze-runtime.md

@@ -0,0 +1,155 @@
+# Analyze an App's Runtime Behavior — Logs, Metrics, Traces, Catalog
+
+## Overview
+
+When you need to understand what an app *actually does* at runtime — why a page is
+slow, which microflow dominates cost, what hits the database, whether an error is
+network or logic — the signals live in four places. This skill is the procedure for
+collecting them and, crucially, **joining them**, because the useful questions cross
+sources that no single tool answers alone.
+
+| Signal | Where | How you get it |
+|--------|-------|----------------|
+| **Logs** (server stack traces + your `LOG` output) | `<projectDir>/.mxcli/runtime.log` | `mxcli run --local` tees it automatically |
+| **Metrics** (throughput, DB counts, sessions, queues) | `/prometheus` on the admin port | `mxcli run --local --metrics` |
+| **Traces** (per-microflow / per-activity spans + timings) | console→`runtime.log`, or an OTLP collector | `mxcli run --local --trace` / `--trace-otlp` |
+| **Model shape** (activities, complexity, refs, XPath) | `.mxcli/catalog.db` (SQLite) | `mxcli … "refresh catalog full"` then `SELECT … FROM CATALOG.*` |
+
+## When to Use This Skill
+
+- A page/microflow is slow and you need to find *where* the time goes.
+- You want to know which entities/queries the app actually hits, and how often.
+- A server-side error shows only a generic dialog in the browser.
+- You're profiling and need a flame chart, or want cost correlated with model shape.
+
+Prerequisite: run the app with the fast local loop — see `run-local.md`. Everything
+below assumes `mxcli run --local` (add the flags noted per signal).
+
+## 1. Logs — the first stop for errors
+
+`run --local` writes the runtime log to `<projectDir>/.mxcli/runtime.log` (override
+`--runtime-log`, `-` disables). It carries JVM stdout/stderr **and** the application
+log — server stack traces, your microflow/nanoflow `LOG` output, and the DB
+synchronization counts at startup.
+
+```bash
+mxcli run --local -p app.mpr
+tail -f .mxcli/runtime.log
+```
+
+Gotchas:
+- **Nanoflow `LOG` lands under the `Client_Nanoflow` node**, not the node name you
+  declared — a filter built around microflow node names silently drops it. `LOG DEBUG`
+  from a nanoflow is dropped server-side (browser console only). See `write-nanoflows.md`.
+- A spike in "Executing N database synchronization command(s)" on an *unchanged* model
+  is a red flag (see the `create or modify` data-loss class of bug).
+
+## 2. Metrics — throughput and database pressure
+
+`--metrics` registers a Prometheus registry, served at
+`http://127.0.0.1:<admin-port>/prometheus` (loopback).
+
+```bash
+mxcli run --local -p app.mpr --metrics
+curl -s http://127.0.0.1:8090/prometheus | grep -E 'connectionbus_|handler_requests|sessions_|taskqueue_'
+```
+
+Useful families: `connectionbus_{selects,inserts,updates,deletes,transactions}_total`
+(database pressure), `handler_requests_total` (throughput), `sessions_*`,
+`taskqueue_*` (background work). Merge any extra registry (otlp/influx/statsd) with
+`--runtime-setting 'Metrics.Registries=[…]'`.
+
+## 3. Traces — where the time goes
+
+`--trace` attaches the bundled OpenTelemetry agent. **Default span filters ship with
+it** (`OpenTelemetry._RuntimeSpanFilters`) because unfiltered per-activity tracing is
+**~10× slower** and produces ~110k spans for one busy transaction — that's a
+flow-*shape* debugging mode, not a timing mode.
+
+```bash
+# console exporter → runtime.log (span names/attrs only)
+mxcli run --local -p app.mpr --trace
+# flame charts: export to a collector (console can't reconstruct call trees/durations)
+mxcli run --local -p app.mpr --trace-otlp http://127.0.0.1:4318
+```
+
+- The **console exporter omits start/end timestamps and parent span IDs** — you get
+  span names + attributes but no call tree and no durations. For real timing/flame
+  charts use `--trace-otlp <endpoint>` (implies `--trace`), which sets the OTLP
+  exporter for you; user-set `OTEL_*` env still wins.
+- `--trace-service NAME` sets `OTEL_SERVICE_NAME` (default the `.mpr` name); use
+  distinct names per app for multi-app correlation. Trace context (W3C `traceparent`)
+  crosses app boundaries automatically over `rest call`.
+- To examine flow *shape* on a small flow, temporarily disable the filters with
+  `--runtime-setting 'OpenTelemetry._RuntimeSpanFilters=[]'`.
+
+## 4. Model shape — the catalog
+
+The catalog is a SQLite database at `.mxcli/catalog.db` describing the model. **Run
+`refresh catalog full`** — plain `refresh catalog` (fast mode) leaves the analytic
+tables (`CATALOG.ACTIVITIES`, `CATALOG.REFS`, `CATALOG.XPATH_EXPRESSIONS`,
+`CATALOG.WIDGETS`) **empty** (a fast-mode query warns "requires refresh catalog full").
+
+```bash
+mxcli -p app.mpr -c "refresh catalog full"
+mxcli -p app.mpr -c "SELECT MicroflowQualifiedName, COUNT(*) activities
+                     FROM CATALOG.ACTIVITIES GROUP BY 1 ORDER BY 2 DESC LIMIT 10"
+```
+
+`CATALOG.ACTIVITIES.Id` is the model GUID the debugger breaks on (see
+`debug-microflows.md`), with the action name and its sequence — a named, ordered
+activity list per microflow. See `catalog-search.md` / `graph-analysis.md` for the
+richer queries.
+
+## 5. The app warehouse — join the signals (external DuckDB)
+
+Each signal alone answers little; the useful questions cross them. Because the catalog
+is a plain database and the app's dev data is Postgres, one engine can join model
+shape + live data + telemetry with **no ETL**. mxcli does **not** embed DuckDB — this
+is a dev-container recipe (dev data, dev telemetry, everything read-only):
+
+```sql
+-- in duckdb, from the project dir, after `refresh catalog full` + a --trace-otlp run
+ATTACH '.mxcli/catalog.db' AS cat (TYPE sqlite, READ_ONLY);
+ATTACH 'dbname=app host=127.0.0.1 user=mendix' AS app (TYPE postgres, READ_ONLY);
+CREATE VIEW spans AS SELECT * FROM read_json_auto('spans.jsonl');
+```
+
+Two joins that are impossible in any single source:
+
+- **Runtime cost × model shape** — span durations per microflow joined to
+  `cat.activities_data` (count) and complexity. Complexity does *not* predict cost: a
+  2-activity flow that delegates can cost more than a 14-activity one. A lint rule
+  can't see that; this join can.
+- **Query time × entity × live rows** — span DB timings joined to `cat` (which entity)
+  and `app` (row counts). This is how you find that the task-queue poller
+  (`system$queuedtask`, owned by no microflow) is the app's largest DB consumer —
+  invisible from any per-microflow view.
+
+Caveats: keep every attachment **read-only** (never a production DB; even locally make
+it explicit), and filter/sample traces first — unfiltered span volume is large.
+
+## Known gap: logs ↔ traces
+
+Runtime log lines carry **no trace id**, so joining logs to traces is a fuzzy
+timestamp join (worst exactly under concurrency). The OTel agent populates the trace
+id in the MDC, but Mendix's log pattern doesn't print it — closing the file-log side
+needs an upstream `%X{trace_id}` change, not mxcli. For collector-side correlation,
+export traces (and logs) via OTLP with `--trace-otlp` so the backend joins them.
+
+## Decision guide
+
+| Question | Reach for |
+|----------|-----------|
+| "Why did this error?" (server-side) | **Logs** (`runtime.log`) |
+| "How much DB / throughput / queue work?" | **Metrics** (`--metrics`) |
+| "Where does the time go in this flow?" | **Traces** (`--trace-otlp` for a flame chart) |
+| "What's the flow's shape / activity list?" | **Catalog** (`refresh catalog full`) or the debugger |
+| "Which cost belongs to which entity/model construct?" | **Warehouse** (catalog × spans × app DB) |
+
+## See Also
+
+- `run-local.md` — the local loop these flags hang off (`--metrics` / `--trace` / `--trace-otlp` / `--runtime-setting` reference).
+- `debug-microflows.md` — interactive breakpoints/stepping when a trace isn't enough.
+- `catalog-search.md`, `graph-analysis.md` — catalog query patterns and dependency analysis.
+- `verify-with-oql.md` — query the running app's data directly.

.claude/skills/mendix/run-local.md

@@ -122,6 +122,7 @@ Launch `run --local` as the **sole** command in its invocation (don't chain a tr
 | `--metrics` | off | Register a Prometheus meter registry at boot; the runtime serves metrics at `http://127.0.0.1:<admin-port>/prometheus` |
 | `--trace` | off | Enable OpenTelemetry tracing (bundled agent, console exporter → the runtime log) with default span filters |
 | `--trace-service` | `.mpr` name | `OTEL_SERVICE_NAME` under `--trace` |
+| `--trace-otlp` | off (console) | Export traces to this OTLP collector endpoint (e.g. `http://127.0.0.1:4318`) instead of the console. Implies `--trace`. Needed for flame charts |
 | `--runtime-setting Key=Value` | — | Merge an extra runtime setting into the boot config (Value parsed as JSON when possible). Repeatable. |
 
 ## Metrics and OpenTelemetry
@@ -159,8 +160,19 @@ with metrics/logs exporters off.
 the microflow-level spans). Override with
 `--runtime-setting 'OpenTelemetry._RuntimeSpanFilters=[…]'`.
 
-**Export to a collector (OTLP) instead of the console:** set the OTEL env yourself
-before running — `--trace` won't override an exporter you've already set:
+**Export to a collector (OTLP) instead of the console:** the console exporter
+omits start/end timestamps and parent span IDs, so call trees and durations can't
+be reconstructed from it — for flame charts, export to an OTLP collector. Pass
+`--trace-otlp <endpoint>` (implies `--trace`); mxcli sets
+`OTEL_TRACES_EXPORTER=otlp`, `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`, and
+`OTEL_EXPORTER_OTLP_ENDPOINT` for you:
+
+```bash
+mxcli run --local -p app.mpr --trace-otlp http://127.0.0.1:4318
+```
+
+You can still set the `OTEL_*` env yourself for full control — `--trace` /
+`--trace-otlp` never override an exporter you've already set:
 
 ```bash
 export OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

CLAUDE.md

@@ -525,7 +525,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati
 - Docker build integration (`mxcli docker build`) with PAD patching (Phase 1)
 - Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--ensure-db` provisions the local Postgres + app database if missing; `--setup` does the non-blocking prerequisites (cache mxbuild+runtime, ensure DB) and exits — `mxcli init` wires it into a Claude Code SessionStart hook so a fresh/reaped web session self-bootstraps, and `docs-site/src/tools/bootstrap-prompt.md` is the empty-repo seed prompt. `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links (repeatable for multi-page sets, one PNG per page) and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md`
 - External browser preview (`mxcli run --hub <url>` + `mxcli tunnel-hub`): the app stays local and reverse-tunnels out over a single 443 connection (embedded chisel) to a static relay, so it is reachable in a browser at a public URL — works from egress-only environments (Claude Code web), verified live through the session's MITM egress proxy. `run --hub` implies `--local`, boots the runtime with `ApplicationRootUrl` set to the assigned URL (so the SPA/`originURI` work under the public origin), resolves the control proxy honouring `NO_PROXY`, and retries forever. `mxcli tunnel-hub --domain <base>` is the **multi-tenant** relay: a registry keyed by prefix/project/solution/branch/worktree (stable URLs on reconnect) fronts many previews at per-subdomain hosts (`[prefix-]project[-branch].<base>`; main collapses to the project) over one 443 with per-subdomain autocert, a registration API (`/api/register|status|deregister|backends`), and a sortable availability overview at `hub.<base>/`. Client identity flags: `--hub-prefix`/`--hub-project`/`--hub-solution`/`--hub-branch`/`--hub-worktree` (project + branch auto-detected). Package: `cmd/mxcli/tunnelhub/`. See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` (slices 3–4)
-- Runtime metrics + settings passthrough (`mxcli run --local --metrics` / `--runtime-setting Key=Value`): `--metrics` registers a Prometheus Micrometer registry at boot (served at `http://127.0.0.1:<admin-port>/prometheus`); `--runtime-setting` merges arbitrary runtime config (e.g. `Metrics.Registries` for otlp/influx/statsd, or `OpenTelemetry._RuntimeSpanFilters`) into mxcli's **single** boot `update_configuration` call — the admin action replaces rather than merges, so folding settings into the one boot call is the only safe way. OTel traces via `--trace` attach the bundled `opentelemetry-javaagent` to the runtime JVM (console exporter → the tee'd runtime log) and ship default `OpenTelemetry._RuntimeSpanFilters` (unfiltered per-activity tracing is ~10× slower); `--trace-service` sets `OTEL_SERVICE_NAME`, and user-set `OTEL_*` env (e.g. an OTLP collector) is respected. See `.claude/skills/mendix/run-local.md`
+- Runtime metrics + settings passthrough (`mxcli run --local --metrics` / `--runtime-setting Key=Value`): `--metrics` registers a Prometheus Micrometer registry at boot (served at `http://127.0.0.1:<admin-port>/prometheus`); `--runtime-setting` merges arbitrary runtime config (e.g. `Metrics.Registries` for otlp/influx/statsd, or `OpenTelemetry._RuntimeSpanFilters`) into mxcli's **single** boot `update_configuration` call — the admin action replaces rather than merges, so folding settings into the one boot call is the only safe way. OTel traces via `--trace` attach the bundled `opentelemetry-javaagent` to the runtime JVM (console exporter → the tee'd runtime log) and ship default `OpenTelemetry._RuntimeSpanFilters` (unfiltered per-activity tracing is ~10× slower); `--trace-service` sets `OTEL_SERVICE_NAME`. The console exporter omits timestamps/parent span IDs (no flame charts), so `--trace-otlp <endpoint>` (implies `--trace`) switches to the OTLP exporter (protocol `http/protobuf`) pointed at a collector; user-set `OTEL_*` env still takes precedence. See `.claude/skills/mendix/run-local.md`
 - OQL query execution against running runtime (`mxcli oql`)
 - Microflow/nanoflow debugger (`mxcli debug`): set breakpoints **by name** (activity resolved from the model), inspect paused flows + variables, step over/into/out, continue — against a `run --local` runtime. Two M2EE planes wired behind one command (admin `enable/disable/status`, app `/debugger/` session); `run --local --debug` enables it at boot. **Nanoflows** are auto-detected (uses the `nanoflow_name` breakpoint param; paused nanoflows are merged from `poll_events`, which `get_paused_microflows` omits). Nanoflow `LOG` output is rewritten to the `Client_Nanoflow` node in the runtime log. See `.claude/skills/mendix/debug-microflows.md` and `docs/11-proposals/PROPOSAL_microflow_debugger.md`
 - Business event services (SHOW/DESCRIBE/CREATE/DROP)

cmd/mxcli/cmd_run.go

@@ -54,8 +54,10 @@ runtime behaviour; it is turned back off on shutdown.
 With --metrics, a Prometheus meter registry is registered at boot and the runtime
 serves metrics at http://127.0.0.1:<admin-port>/prometheus. With --trace, the
 bundled OpenTelemetry agent is attached (spans -> the runtime log via the console
-exporter) with default span filters (unfiltered tracing is ~10x slower). Use
---runtime-setting Key=Value (repeatable) to merge any other runtime setting into
+exporter) with default span filters (unfiltered tracing is ~10x slower). The
+console exporter omits timestamps and parent span IDs, so for flame charts pass
+--trace-otlp <endpoint> to export to an OTLP collector (e.g. http://127.0.0.1:4318)
+instead. Use --runtime-setting Key=Value (repeatable) to merge any other runtime setting into
 the boot config (the admin config action replaces rather than merges, so mxcli
 folds these into its single boot call), e.g. a different Metrics.Registries type or
 custom OpenTelemetry span filters.
@@ -120,6 +122,10 @@ Examples:
 		runtimeSettings, _ := cmd.Flags().GetStringArray("runtime-setting")
 		trace, _ := cmd.Flags().GetBool("trace")
 		traceService, _ := cmd.Flags().GetString("trace-service")
+		traceOTLP, _ := cmd.Flags().GetString("trace-otlp")
+		if traceOTLP != "" {
+			trace = true // --trace-otlp implies --trace
+		}
 
 		opts := docker.LocalRunOptions{
 			ProjectPath:        projectPath,
@@ -148,6 +154,7 @@ Examples:
 			RuntimeSettings:    runtimeSettings,
 			Trace:              trace,
 			TraceService:       traceService,
+			TraceOTLP:          traceOTLP,
 			DB: docker.DBConfig{
 				Host:     dbHost,
 				Name:     dbName,
@@ -196,5 +203,6 @@ func init() {
 	runCmd.Flags().StringArray("runtime-setting", nil, "Extra runtime setting Key=Value merged into the boot configuration (Value parsed as JSON when possible), e.g. --runtime-setting 'OpenTelemetry._RuntimeSpanFilters=[\"Loop\",\"Gateway\"]'. Repeatable.")
 	runCmd.Flags().Bool("trace", false, "Enable OpenTelemetry tracing: attach the bundled agent (console exporter → the runtime log) and apply default span filters (unfiltered tracing is ~10x slower)")
 	runCmd.Flags().String("trace-service", "", "OTEL_SERVICE_NAME under --trace (default: the .mpr name)")
+	runCmd.Flags().String("trace-otlp", "", "Export traces to this OTLP collector endpoint (e.g. http://127.0.0.1:4318) instead of the console — needed for flame charts (the console exporter omits timestamps/parent IDs). Implies --trace")
 	rootCmd.AddCommand(runCmd)
 }