Spec: canonical JSON Schema service for validation and shared tooling

[BorisTyshkevich]

Implementation ready in PR #222; awaiting human merge.

Problem

The saved-query Spec editor now has JSON parsing, continuous diagnostics, atomic Save, and an app-owned path-validator registry from #212. The current validation model is intentionally small:

CORE_SPEC_VALIDATORS
createSpecValidatorRegistry()
register(path, validate)

That was sufficient to validate the first core fields, but it is not a scalable source of truth for the growing Spec surface:

• panel types and their discriminated configuration;
• panel.fieldConfig.columns metadata;
• dashboard roles and filter sources;
• transformations, links, layout, and refresh settings;
• future composite visual objects such as KPI tuples, gauges, candlesticks, confidence bands, box plots, ranges, Gantt tasks, heatmaps, networks, and status values.

Hand-written validators alone do not describe:

• which properties are valid at a JSON path;
• property and enum documentation;
• defaults and examples;
• discriminated panel.cfg branches;
• reusable definitions and references;
• information required by CodeMirror completion.

Without one canonical schema, editor diagnostics, Save validation, import validation, runtime panel checks, examples, and completion will drift.

Goal

Add a pure, reusable Spec schema service backed by a canonical JSON Schema Draft 2020-12 document for query.spec.

The service must provide:

1. structural and static semantic validation;
2. stable path-based diagnostics;
3. schema lookup and branch resolution for editor tooling;
4. a feature-validator layer for rules that JSON Schema cannot express;
5. reusable validation for Save, import, external Spec writers, runtime consumers, and repository examples.

This issue does not implement CodeMirror completion. Completion is a dependent issue and consumes the schema service defined here.

Current base

This issue builds on:

• Saved-query format v2: separate SQL from an extensible Spec document #211: canonical saved-query shape {id, sql, specVersion, spec};
• Workbench: add separate SQL and Spec JSON editor modes #212: separate SQL and Spec editors, continuous diagnostics, exact path arrays, and atomic Save;
• the existing src/core/spec-draft.js parsing and validator registry;
• the existing src/editor/spec-editor.js diagnostic range mapping.
• docs/saved-query-spec-json-schema.md
• docs/saved-query-spec-v1.schema.json
• docs/visualization-spec-authoring-guide.md

The user-editable document remains only:

query.spec

The schema must not expose or validate the surrounding saved-query fields:

id
sql
specVersion
export envelope
compatibility mirrors
runtime state

Canonical schema

Add a machine-readable schema:

schemas/query-spec-v1.schema.json

Required header:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://altinity.com/schemas/altinity-sql-browser/query-spec-v1.schema.json",
  "title": "Altinity SQL Browser saved-query Spec v1",
  "type": "object"
}

The schema is the source of truth for known static structure. It must cover at least the fields implemented when this issue lands:

name
description
favorite
view
panel
dashboard
panel.cfg
panel.fieldConfig

Panel branches use a discriminator-like type field with const values:

{
  "oneOf": [
    {
      "properties": {
        "type": { "const": "table" }
      },
      "required": ["type"]
    },
    {
      "properties": {
        "type": { "const": "logs" },
        "time": { "$ref": "#/$defs/columnName" },
        "msg": { "$ref": "#/$defs/columnName" },
        "level": { "$ref": "#/$defs/columnName" }
      },
      "required": ["type"]
    }
  ],
  "x-altinity-discriminator": "type"
}

Every implemented panel feature must update the canonical schema in the same PR that adds or changes its Spec contract.

Required existing-panel baseline

The canonical schema must fully describe every panel arm already implemented by the application. Future/planned panel contracts may also be present, but tooling must be able to distinguish them from renderers available in the current build.

Every panel.cfg branch must carry:

title                         human-facing panel name
description                   concise authoring documentation
x-altinity-status             implemented | planned
x-altinity-snippet            schema-owned starter structure

Completion offers implemented panel types by default. A planned branch is documentation and forward-compatible validation only until its renderer is registered.

There is no separate panel.name field:

spec.name                                           panel/tile title
panel.fieldConfig.columns.<column>.displayName      rendered field label
panel.cfg.type                                      visualization type

Chart family: bar | hbar | line | area | pie

The existing chart family keeps its current zero-based result-column index contract:

{
  "type": "line",
  "x": 0,
  "y": [1, 2],
  "series": 3
}

Static schema rules:

• type, x, and y are required;
• x is a non-negative integer;
• y is a non-empty array of unique non-negative integers;
• series is a non-negative integer or null;
• Pie requires exactly one measure: y.maxItems = 1;
• unknown extra cfg fields remain accepted and preserved.

x, each y item, and non-null series carry:

{
  "x-altinity-completion": {
    "source": "resultColumnIndexes"
  }
}

The completion source displays index, name, and ClickHouse type while inserting only the integer index.

Result-aware feature validation remains responsible for:

• index bounds;
• X/measure/series type suitability;
• series !== x;
• measure indexes not conflicting with X/series where the renderer forbids it;
• saved panel.key mismatch and role re-derivation.

Table: table

{
  "type": "table"
}

The table branch has no required cfg details. Sort order, widths, and other current table state remain surface-local unless a later Spec issue defines persistent fields.

Logs: logs

{
  "type": "logs",
  "time": "event_time",
  "msg": "message",
  "level": "level"
}

time, msg, and level are optional exact result-column names. They use the shared columnName definition and its resultColumns completion source.

Static schema validates that supplied roles are non-empty strings. Result-aware validation checks that columns exist. time and msg are load-bearing; a missing level degrades to no level coloring, matching the existing renderer.

Markdown text: text

{
  "type": "text",
  "content": "# Heading\n\nMarkdown content."
}

Only type is required. content is optional, defaults to "" for documentation/completion, and must be a string when present. This matches the existing runtime normalization of missing content to an empty string.

The schema validates the JSON field, not Markdown syntax. The safe Markdown parser/renderer remains authoritative for the supported subset and injection safety.

Existing-panel validation matrix

Panel branch	Static JSON Schema	Result/context validator	Completion source
Chart family	type, index shapes, unique/non-empty y, Pie one-measure limit	bounds, ClickHouse type suitability, role conflicts, panel.key mismatch	resultColumnIndexes
Table	branch/discriminator only	none currently	schema title/docs/snippet
Logs	optional non-empty role names	result-column resolution and log-shape viability	resultColumns
Text/Markdown	optional string content	safe Markdown rendering remains defensive	schema property/snippet
Unknown future type	non-empty string plus extension preservation	unsupported-type diagnostic and fallback	not offered as an implemented type

Extension-preservation rule

Saved-query v2 requires unknown fields to survive all read/write paths. The schema must therefore validate known fields without rejecting extension data.

Rules:

• the root Spec allows unknown properties;
• panel, dashboard, fieldConfig, and feature objects allow unknown properties unless a future version explicitly changes that contract;
• known fields with an invalid value are errors;
• unknown fields are accepted and preserved;
• unknown panel.cfg.type values remain storable for forward compatibility, but are not considered renderable by the current build.

Unknown-type handling needs two layers:

1. schema storage validation accepts an object with a non-empty string type;
2. the runtime panel registry reports that the type is unsupported and applies its documented fallback behavior.

The discriminated union must include an explicit forward-compatible fallback branch rather than relying on a failing oneOf. For example:

{
  "allOf": [
    {
      "required": ["type"],
      "properties": {
        "type": { "type": "string", "minLength": 1 }
      }
    },
    {
      "oneOf": [
        {
          "properties": { "type": { "const": "table" } }
        },
        {
          "properties": { "type": { "const": "logs" } }
        },
        {
          "properties": {
            "type": {
              "not": { "enum": ["table", "logs"] }
            }
          }
        }
      ]
    }
  ]
}

The real enum contains every implemented type. The fallback matches only unknown types, so known branches remain unambiguous.

Do not use additionalProperties: false as a typo detector in extensible namespaces.

Build-time compilation

Use a mature Draft 2020-12 validator rather than implementing JSON Schema semantics by hand.

Recommended implementation:

• add ajv as a dev dependency;
• compile the canonical schema with Ajv 2020 standalone generation;
• ship only the generated validator in the browser bundle;
• do not add Ajv to runtime dependencies;
• keep the raw schema available to schema lookup and completion code.

Suggested files:

schemas/query-spec-v1.schema.json
build/compile-spec-schema.mjs
src/generated/query-spec-v1-validator.js

The generator must:

• run deterministically;
• register the x-altinity-* annotation keywords so strict schema checking remains enabled;
• fail on unresolved $ref values or an invalid schema;
• emit no network-loading code;
• produce stable output suitable for a drift check.

Add a command such as:

{
  "scripts": {
    "generate:spec-schema": "node build/compile-spec-schema.mjs"
  }
}

npm test and npm run build must fail when the generated validator is stale relative to the canonical schema.

Equivalent build-time compilation is acceptable, provided the production artifact has no general JSON Schema validator dependency.

Pure schema service

Add:

src/core/spec-schema.js

Suggested interface:

createSpecSchemaService({
  schema,
  validateCompiled,
}) -> {
  schema,
  validate(value),
  schemaAtPath({ root, path }),
  propertiesAtPath({ root, path }),
  annotationsAtPath({ root, path }),
}

validate(value)

Returns normalized diagnostics:

{
  path: Array<string | number>,
  severity: 'error' | 'warning' | 'info',
  code: string,
  message: string,
  keyword?: string,
}

Schema-validation diagnostics are blocking errors unless a specific schema annotation says otherwise.

Stable code mapping:

type                  schema-invalid-type
required              schema-required
const                 schema-invalid-constant
enum                  schema-invalid-enum
minimum/maximum       schema-number-range
minLength/pattern     schema-invalid-string
minItems/maxItems     schema-array-size
uniqueItems           schema-array-duplicate
oneOf                 schema-invalid-variant
$ref resolution       schema-internal-reference

Do not expose Ajv-specific error object shapes to the rest of the application.

Diagnostic paths

Use the exact path-array model already established by #212.

Examples:

['panel', 'cfg', 'type']
['panel', 'fieldConfig', 'columns', 'latency', 'decimals']
['dashboard', 'filters', 0, 'param']

Required-property errors point to the missing child path when possible:

['panel', 'cfg', 'type']

not only to the containing object:

['panel', 'cfg']

Keys containing dots remain one path segment.

Schema path resolution

schemaAtPath() and propertiesAtPath() are read-only schema introspection APIs for completion and documentation.

They must resolve the subset used by the canonical schema:

$ref
allOf
oneOf
anyOf
if / then / else
properties
patternProperties
additionalProperties
items
prefixItems
const
enum
default
examples

For a discriminated oneOf, the current Spec value selects the branch:

{
  "type": "candlestick"
}

must resolve only the candlestick branch rather than merging unrelated panel fields.

When the discriminator is absent or incomplete, the service may return multiple candidate branches, but it must expose the common properties and the discriminator first.

The service is pure:

• no DOM;
• no CodeMirror imports;
• no application state;
• no fetch;
• no SQL execution.

Feature/runtime validators

JSON Schema handles static document correctness. It cannot validate facts that depend on query results, ClickHouse types, or application state.

Retain a feature-validator registry, but make it part of one validation service instead of a parallel ad hoc model.

Suggested interface:

createSpecValidationService({ schemaService }) -> {
  register(path, validate),
  validate(spec, context?),
}

validate() runs:

1. canonical schema validation;
2. registered feature validators only when schema validation produced no blocking ancestor error for their path.

Feature validator context may include:

{
  resultColumns,
  resultRows,
  sqlDraft,
  savedQueries,
  serverVersion,
}

Examples of rules that remain feature/runtime validation:

• configured result column does not exist;
• KPI scalar is non-numeric;
• named tuple is missing required member value;
• gauge min >= max when bounds come from runtime tuple members;
• candlestick low > high;
• confidence lower > upper;
• box-plot quantiles are out of order;
• Gantt dependency cycles;
• network edge references an unknown node;
• an authored SQL FORMAT clause conflicts with a panel-owned result format.

The canonical schema should describe the static shape of these configurations, but must not pretend to validate result-dependent correctness.

Consumers

1. Spec editor diagnostics

Replace CORE_SPEC_VALIDATORS as the primary static validator with the schema validation service.

The editor evaluation order remains:

JSON syntax
→ canonical schema
→ feature/runtime validators
→ normalized diagnostics

Invalid JSON still skips all semantic validation and remains editable.

2. Atomic Save

Linked Save must synchronously rerun schema and feature validation before persistence.

A stale debounced editor result must never authorize Save.

Any blocking error persists neither SQL nor Spec.

3. File import, Replace, and Append

Validate each supported query after v1-to-v2 upgrade and before mutating the Library.

Behavior:

• unsupported envelope or Spec versions retain their current explicit rejection;
• schema-invalid supported Specs are rejected with query id/index plus the first diagnostic path;
• the operation is atomic: invalid input does not partially replace or append entries;
• unknown extension fields remain accepted.

Example message:

Query "latency-kpi": panel.fieldConfig.columns.latency.decimals must be an integer.

4. External Spec writers

Library pencil, favorite toggle, Panel controls, and future dashboard controls already patch the canonical Spec.

Their shared patch helper must:

1. clone the complete Spec;
2. apply the known patch;
3. run schema validation;
4. persist only when the patched result is valid;
5. preserve unrelated and unknown fields.

Existing #212 invalid-JSON conflict handling remains authoritative for open drafts.

5. Runtime panel and dashboard consumers

Panel and dashboard code can use static schema validation instead of repeating primitive type and enum checks.

Examples:

• panelCfgValid() retains result-shape checks but delegates static cfg shape to the schema service;
• dashboard role partitioning uses schema-normalized role values;
• renderers receive concise diagnostics rather than raw validator-library errors.

Do not remove renderer-level defensive checks. Imported storage can be corrupted, older builds may have written unsupported values, and runtime result data remains untrusted.

6. Repository examples and generated Libraries

Add a test/helper that validates:

• every checked-in example Library;
• generated Library output;
• Spec examples used in documentation;
• test fixtures claiming to be valid.

This prevents documentation and sample files from drifting from the actual application contract.

Schema annotations for tooling

The validator treats these as annotations only:

x-altinity-discriminator
x-altinity-completion
x-altinity-key-completion
x-altinity-snippet
x-altinity-order
x-altinity-deprecated
x-altinity-status

Example:

{
  "$defs": {
    "columnName": {
      "type": "string",
      "minLength": 1,
      "description": "Exact top-level ClickHouse result-column name.",
      "x-altinity-completion": {
        "source": "resultColumns"
      }
    }
  }
}

The schema-service issue only preserves and exposes these annotations. The dependent CodeMirror issue interprets them. x-altinity-status distinguishes implemented branches from planned design contracts so completion does not advertise unavailable panels.

For chart index fields, the schema uses resultColumnIndexes; for Logs and other name-based roles it uses resultColumns.

Normalization

Validation and normalization remain separate operations.

Schema validation must not silently rewrite the Spec.

Continue to normalize only explicitly settled fields during successful Save, for example:

trim name
trim/remove blank description
normalize implemented enum aliases only when a migration explicitly defines them

Do not apply JSON Schema default values by mutating the document. Defaults exist for documentation, completion, and runtime read helpers.

Files

Expected changes:

schemas/query-spec-v1.schema.json
build/compile-spec-schema.mjs
src/generated/query-spec-v1-validator.js
src/core/spec-schema.js
src/core/spec-draft.js
src/core/saved-io.js
src/core/saved-query.js
src/core/panel-cfg.js
src/state.js
src/ui/app.js
tests/unit/spec-schema.test.js
tests/unit/spec-draft.test.js
tests/unit/saved-io.test.js
tests/unit/state.test.js
example/schema validation tests
package.json
package-lock.json

Equivalent factoring is acceptable.

Tests

Canonical schema

• validates against the Draft 2020-12 meta-schema;
• every local $ref resolves;
• $id and Spec version remain stable;
• annotation keywords are accepted in strict compilation;
• generated validator is current and deterministic.

Static validation

• accepts a minimal valid Spec;
• accepts all known valid panel branches;
• validates the implemented chart/table/logs/text baseline exactly;
• accepts Text without content and rejects non-string content;
• rejects Pie configs with more than one y measure;
• validates chart index shapes while leaving result-dependent bounds to feature validators;
• accepts unknown extension fields at every extensible level;
• reports wrong known-field types;
• reports missing required discriminators;
• reports invalid enum and range values;
• maps required errors to missing-child paths;
• preserves dotted keys and numeric array indexes as exact path segments;
• collapses oneOf failures into concise diagnostics rather than exposing every branch error.

Schema lookup

• resolves nested $ref values;
• selects a discriminated branch from the current Spec;
• returns common candidates when the discriminator is absent;
• resolves array-item and dynamic-object value schemas;
• exposes descriptions, defaults, examples, branch titles, x-altinity-status, snippets, and other x-altinity-* annotations;
• exposes resultColumnIndexes annotations on chart roles and resultColumns annotations on Logs roles.

Integration

• Spec editor receives schema diagnostics;
• Save synchronously revalidates and remains atomic;
• file Replace/Append reject invalid Specs without partial mutation;
• external writers do not persist an invalid patched Spec;
• panel runtime checks still catch result-dependent errors;
• panelCfgValid() or its successor delegates static chart/logs/text shape checks to the schema service without removing renderer defenses;
• valid example Libraries and generated output pass;
• invalid fixtures produce stable path-based messages.

Regression

• invalid JSON remains editable and blocks persistence;
• unknown fields survive Save/import/share/external patches;
• current core validator behavior remains covered;
• no SQL request occurs during schema validation;
• production dependencies do not gain a general JSON Schema validator;
• coverage gates and build pass.

Acceptance criteria

• A canonical Draft 2020-12 schema for query.spec is checked in.
• The schema validates known structure while accepting and preserving unknown extensions.
• All currently implemented panel types (bar, hbar, line, area, pie, table, logs, text) have explicit branches, titles, descriptions, snippets, and static constraints.
• Text content is optional with a documented empty-string default; Pie accepts exactly one measure.
• Chart index fields and Logs name fields expose the correct dynamic-completion annotations.
• Implemented and planned panel branches are distinguishable so tooling does not advertise unavailable renderers.
• A pure schema service exposes validation and schema-at-path lookup.
• Production uses a build-time-compiled validator with no runtime Ajv dependency.
• Diagnostics use stable codes and exact path arrays.
• The existing feature-validator registry is integrated for result- and context-dependent rules.
• Spec editor validation and atomic Save use the schema service.
• Import/Replace/Append validate before mutating the Library.
• External Spec writers validate their patched result before persistence.
• Runtime panel/dashboard code can reuse static schema checks without losing defensive result validation.
• Checked-in examples and generated Libraries are schema-validated in tests.
• The service exposes the annotations and branch information required by a dependent CodeMirror completion issue.
• Unknown fields survive every supported read/write path.
• Coverage gates and build pass.

Non-goals

• CodeMirror completion UI; tracked by the dependent completion issue.
• A visual form builder for Spec.
• Applying schema defaults by rewriting user JSON.
• Rejecting unknown extension fields.
• Replacing result-dependent or server-version-dependent feature validation with JSON Schema.
• Supporting remote $ref loading.
• Editing id, SQL, specVersion, export envelopes, compatibility mirrors, or runtime state.