Skip to content

refactor(codegen): split plan for main_zig.zig + PoC extraction (RFC for #183) - #193

Closed
apotema wants to merge 3 commits into
mainfrom
refactor/183-main-zig-investigation
Closed

apotema wants to merge 3 commits into
mainfrom
refactor/183-main-zig-investigation

Conversation

@apotema

@apotema apotema commented May 25, 2026

Copy link
Copy Markdown
Contributor

Summary

src/main_zig.zig is the 4,692-line code generator that every backend × platform combo flows through. A full split is high-risk: bit-identical generated output must be preserved for every example, and there are ~18 template slots wired through a single orchestrator.

This PR is an RFC for the cut plan plus one proof-of-concept extraction. The rest of the cuts are deferred until reviewers vet the approach. Refs #183.

What's in this PR

  1. docs/REFACTOR-PLAN-main-zig.md — section-by-section map of main_zig.zig, proposed module layout (src/codegen/{scan, idents, validate, blocks/*, lifecycle/*, preview, main_template}.zig), dependency graph, per-section risk assessment, and ordering for future extraction PRs.
  2. src/codegen/scan.zig (875 lines) — the PoC extraction. Holds the AST-walk discovery helpers (PluginEvent, PluginFlowDecls, PluginPinStyle, PluginCoercion, discoverPluginEvents, discoverPluginFlowDecls, dedupePinStyles, sanitizePluginIdent, pathToIdent) plus the existing pathToIdent tests. Chosen because it's pure data collection — no template state, no codegen-orchestrator coupling, well-defined public API already used externally.
  3. src/main_zig.zig (4,692 → 3,880 lines, -812) — re-exports the moved symbols so root.zig and test/tests.zig keep their existing imports unchanged. Pure move + import; no call sites moved, no logic touched.
  4. scripts/gen_all_examples.sh — regenerates every bundled example so before/after diffs can verify the bit-identical contract.

Verification

  • zig build clean.
  • zig build test exits 0 (same as origin/main; "failed command" log lines are pre-existing diagnostic-output tests, not test failures).
  • Bit-identical generated output across all 9 bundled examples (raylib, sokol, sokol_imgui, null, plugin-controllers, flows-smoke, asset-streaming-smoke, bgfx, wgpu):
    find /tmp/asm-before -name "main.zig" -not -path "*/deps/*" | while read f; do
      after=${f/asm-before/asm-after}
      cmp -s "$f" "$after" || echo "DIFF: $f"
    done
    # → zero diffs
    
    The only diff -r drift across the entire generated tree is the generated_at ISO-8601 timestamp inside flow_catalog.json (expected and unrelated).

Constraints honored

  • No logic rewritten — pure move + import.
  • Public API unchanged: every @import("main_zig.zig").Plugin* / .discover* site keeps working via re-exports.
  • No further extractions attempted.
  • No backend templates touched.

Asks for review

  1. Is the proposed module layout the right shape? Particularly: do you want src/codegen/blocks/*.zig per-slot or one blocks.zig? Per-backend lifecycle/* vs one file?
  2. Is the ordering sensible? Plan lists 10 steps; PoC is step 1. Steps 2-9 each ship as their own PR with the same bit-identical guarantee.
  3. Any preview/orchestrator state I'm under-estimating? The risk table flags preview.zig (~1200 lines of inline templates) and lifecycle/* as medium-to-high. The orchestrator itself stays last.

If the plan looks wrong, the PoC reverts cleanly (one re-export shim, one new file).

Test plan

  • Reviewer skims docs/REFACTOR-PLAN-main-zig.md and reacts to the proposed module layout / ordering.
  • Reviewer spot-checks src/codegen/scan.zig against the deleted regions of src/main_zig.zig to confirm pure move.
  • CI runs zig build + zig build test.
  • Reviewer optionally re-runs bash scripts/gen_all_examples.sh /tmp/x on this branch vs origin/main to reproduce the bit-identical check.

DRAFT — do not merge until plan is reviewed.

…#183)

main_zig.zig is the 4,692-line code generator every backend × platform
combo flows through. Splitting it carries real risk: any drift in
emission order, allocator lifetime, or template-slot string content
breaks bit-identical output for every downstream backend, and there are
~18 named slots wired through one orchestrator.

This PR does NOT attempt the full split. It lands:

1. `docs/REFACTOR-PLAN-main-zig.md` — section-by-section map of the
   current file, proposed module layout (`src/codegen/{scan, idents,
   validate, blocks/*, lifecycle/*, preview, main_template}.zig`),
   dependency graph, per-section risk assessment, and the ordering for
   future extraction PRs.
2. `src/codegen/scan.zig` — the lowest-risk extraction from the plan:
   the AST-walk discovery helpers (`PluginEvent`, `PluginFlowDecls`,
   `PluginPinStyle`, `PluginCoercion`, `discoverPluginEvents`,
   `discoverPluginFlowDecls`, `dedupePinStyles`, `sanitizePluginIdent`,
   `pathToIdent`) plus their existing tests. Pure data collection with
   no template-state coupling and a stable external API already used
   from `root.zig` + `test/tests.zig`.
3. `main_zig.zig` re-exports the moved symbols so every existing
   `@import("main_zig.zig").Plugin*` / `.discover*` site keeps working
   unchanged. No call sites moved; no logic touched. Net: 4,692 → 3,880
   lines (-812).
4. `scripts/gen_all_examples.sh` — regenerates every bundled example
   (raylib, sokol, sokol_imgui, null, plugin-controllers, flows-smoke,
   asset-streaming-smoke, bgfx, wgpu) so before/after diffs can verify
   the bit-identical contract.

Verification: `zig build test` exits 0 (same as origin/main). Across
all 9 examples, every generated `main.zig` is byte-identical pre/post
extraction; the only `diff -r` drift is the `generated_at` timestamp
inside `flow_catalog.json`.

Draft: requesting human review of the cut plan before any further
extraction PRs. Refs #183.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request initiates a refactoring plan to split the large src/main_zig.zig file by extracting discovery and scanning helpers into a new submodule, src/codegen/scan.zig, while maintaining backward compatibility via re-exports. It also introduces a verification script to ensure bit-identical output generation across examples. The review feedback highlights critical memory leak risks in the newly extracted scanning code where partial allocations are not properly cleaned up with block-scoped errdefers on failure. Additionally, it recommends replacing std.debug.print with standard logging facilities (std.log.err) for user-facing errors in CLI contexts.

Comment thread src/codegen/scan.zig
Comment on lines +187 to +191
try entries.append(allocator, .{
.plugin_import_name = try allocator.dupe(u8, module_name),
.plugin_sanitized = try allocator.dupe(u8, sanitized),
.event_name = try allocator.dupe(u8, event_name),
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If any of the subsequent allocator.dupe allocations or the entries.append call fail, the previously allocated strings in the current iteration will be leaked. Using block-scoped errdefers ensures that these temporary allocations are cleaned up if the iteration fails before ownership is transferred to entries.

            const duped_import_name = try allocator.dupe(u8, module_name);
            errdefer allocator.free(duped_import_name);
            const duped_sanitized = try allocator.dupe(u8, sanitized);
            errdefer allocator.free(duped_sanitized);
            const duped_event_name = try allocator.dupe(u8, event_name);
            errdefer allocator.free(duped_event_name);

            try entries.append(allocator, .{
                .plugin_import_name = duped_import_name,
                .plugin_sanitized = duped_sanitized,
                .event_name = duped_event_name,
            });

Comment thread src/codegen/scan.zig
Comment on lines +496 to +504
const init_src = ast.getNodeSource(member_init);
const constructs_value = extractConstructsString(allocator, init_src);
try flow_nodes_out.append(allocator, .{
.module_import_path = try allocator.dupe(u8, module_import_path),
.module_sanitized = try allocator.dupe(u8, module_sanitized),
.node_name = try allocator.dupe(u8, member_name),
.is_script = is_script,
.constructs = constructs_value,
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If any of the allocator.dupe calls or the append call fail, constructs_value (which is allocated by extractConstructsString) and any previously allocated strings in the current iteration will be leaked. Block-scoped errdefers should be used to prevent these memory leaks on allocation failure.

                const init_src = ast.getNodeSource(member_init);
                const constructs_value = extractConstructsString(allocator, init_src);
                errdefer if (constructs_value) |c| allocator.free(c);

                const duped_path = try allocator.dupe(u8, module_import_path);
                errdefer allocator.free(duped_path);
                const duped_sanitized = try allocator.dupe(u8, module_sanitized);
                errdefer allocator.free(duped_sanitized);
                const duped_name = try allocator.dupe(u8, member_name);
                errdefer allocator.free(duped_name);

                try flow_nodes_out.append(allocator, .{
                    .module_import_path = duped_path,
                    .module_sanitized = duped_sanitized,
                    .node_name = duped_name,
                    .is_script = is_script,
                    .constructs = constructs_value,
                });

Comment thread src/codegen/scan.zig Outdated
const append = struct {
fn f(b: *[256]u8, idx: *usize, bytes: []const u8) void {
if (idx.* + bytes.len > b.len) {
std.debug.print("labelle: path too long for identifier (max {d} chars): too many escaped chars\n", .{b.len});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the general rules, debug-specific print functions like std.debug.print should not be used for user-facing warnings or errors in CLI tools. Standard logging facilities (such as std.log.err) should be used instead.

                std.log.err("labelle: path too long for identifier (max {d} chars): too many escaped chars", .{b.len});
References
  1. In CLI tools, use standard logging facilities (e.g. std.log.warn) or write to stderr for user-facing warnings, rather than using debug-specific print functions (e.g. std.debug.print).

Three findings from gemini on #193, all pre-existing bugs the PoC
extraction surfaced. Fixed only in src/codegen/scan.zig — main_zig.zig
already re-exports these symbols (no duplicate copies to keep in sync).

- :191 — errdefer per dupe before append, prevents leak when one
  step in the chain fails.
- :504 — same shape around extractConstructsString + 3 dupes; the
  optional constructs allocation gets its own errdefer guard. Also
  applied to the parallel pin_styles_out and coercions_out appends
  in the same loop for consistency.
- :757 — std.debug.print -> local stderrPrint helper (matches the
  writeStderr pattern in main.zig / cache_cmd.zig; std.log.err can't
  be used because it trips the test-runner log interceptor, same
  rationale documented in src/scene_manifest.zig).

Bit-identical generated output across all bundled examples (8/8 of
the working ones — sokol_imgui was already failing pre-fix and is
unrelated). zig build test green: 506/510 (4 skipped), same as
baseline.
@apotema

apotema commented May 25, 2026

Copy link
Copy Markdown
Contributor Author

Now under umbrella PR #202

The content from this PR is already on the umbrella integration branch feat/183-split-main-zig (which #202 targets to main). The umbrella aggregates the whole main_zig.zig split — every leaf in the stack and the eventual mixin conversion pass.

You can either:

  • Review + merge this PR individually, then the umbrella's diff shrinks as the leaves below land
  • Or skip individual review and review the whole split via Umbrella: split src/main_zig.zig (#183) #202 when it's ready — in which case this PR closes (content lands via the umbrella merge)

Either way the actual code on main ends up identical.

Round-N follow-up to a02c6cd. The earlier pass fixed the dupe-chain
sites that gemini flagged inline; this pass walks every remaining
allocator.dupe loop and every readFileAlloc call in the file, plus the
toOwnedSlice cascade in discoverPluginFlowDecls.

- PluginEvent doc comment (line 46) updated: was "both strings are
  heap-allocated dupes" but the struct has three string fields.
- discoverEventsFromRoot readFileAlloc (line ~165) now propagates
  error.OutOfMemory instead of swallowing every error as "this module
  contributes nothing" — same applied to the plugin pass (line ~663)
  and the script pass (line ~693) in discoverPluginFlowDecls.
- discoverPluginFlowDecls return chain (line ~711): three back-to-back
  toOwnedSlice calls — each one detaches its ArrayList from the
  enclosing errdefer, so a mid-sequence OOM leaked the previously
  detached slice. Stage each slice into a local with a slice-shaped
  errdefer so OOM stays clean across all three.

Dismissed as already fixed in a02c6cd:
- per-dupe errdefer at lines 209-220 (events loop)
- per-dupe errdefer at lines 525-547 (flow_nodes loop)
- per-dupe errdefer at lines 549-561 (pin_styles loop)
- per-dupe errdefer at lines 573-585 (coercions loop)
- std.debug.print at line 814 (already routes through stderrPrint
  helper a02c6cd added at the top of the file)

zig build test exit 0, 346/350 passing (4 skips, unchanged from
baseline).
scripts/gen_all_examples.sh: 8/8 main.zig files bit-identical to
baseline (sokol_imgui pre-existing soft-fail unchanged).
apotema added a commit that referenced this pull request May 25, 2026
…ig (part of #183)

Per the plan in #193. Moved LoadStyle + emitResourceLoad to
src/codegen/blocks/resource_loader.zig (113 lines). Re-exported from
main_zig.zig so callers in both lifecycle paths (buildSetupCode + the
callback init builder) keep their existing references.

Depends on idents.zig from #195 (extWithoutDot, isValidZigIdentifier).

Bit-identical generated output across all 8 working bundled examples
(scripts/gen_all_examples.sh green; sokol_imgui pre-existing skip
because labelle-imgui sibling not present).

Refs: #183
Builds on: #195

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
apotema added a commit that referenced this pull request May 25, 2026
…ig (part of #183)

Per the plan in #193. Moved writeSceneAssetManifests and
writeSceneInitialStateManifests to src/codegen/blocks/scene_manifests.zig
(137 lines). Re-exported from main_zig.zig so callers don't change.

Depends on idents.zig from #195 (writeZigString, pathToIdent).

Bit-identical generated output across all 8 working bundled examples
(scripts/gen_all_examples.sh green).

Refs: #183
Builds on: #195

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
apotema added a commit that referenced this pull request May 25, 2026
….zig (part of #183)

Per the plan in #193. Moved 5 plugin-registry block writers
(controllers, events, flow_nodes, pin_styles, coercions) to
src/codegen/blocks/plugin_registries.zig (353 lines). Re-exported from
main_zig.zig so the orchestrator's call sites stay unchanged.

Depends on scan types (PluginEvent, PluginFlowNode, PluginPinStyle,
PluginCoercion) from #193 and ident helpers from #195.

Bit-identical generated output across all 8 working bundled examples
(scripts/gen_all_examples.sh green).

Refs: #183
Builds on: #193, #195

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
apotema added a commit that referenced this pull request May 25, 2026
…(part of #183)

Fourth extraction per the plan in #193. Moved
writeImageBackendWiring/writeAudioBackendWiring/writeFontBackendWiring
to src/codegen/blocks/asset_wiring.zig (377 lines of pure emit, no state).
Re-exported from main_zig.zig so callers don't change.

Bit-identical generated output across all 8 working bundled examples
(scripts/gen_all_examples.sh green; sokol_imgui soft-fails on missing
labelle-imgui sibling, same as baseline).

Refs: #183
Builds on: #195

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
apotema added a commit that referenced this pull request May 25, 2026
…part of #183)

Per the plan in #193. Moved buildCallbackInitCode +
buildImmersiveEntryCode + buildCallbackCleanupCode (sokol/wasm/Android
callback path) to src/codegen/lifecycle/callback.zig (252 lines).
Re-exported from main_zig.zig so callers in the orchestrator stay
unchanged.

HIGH-risk extraction per the plan -- preserves call order across the
preview-mode wiring constants and the wasm/Android conditional
branches. Verified bit-identical across all 8 working bundled
examples (scripts/gen_all_examples.sh).

The asset-wiring + resource-loader helpers (writeImageBackendWiring,
writeAudioBackendWiring, writeFontBackendWiring, emitResourceLoad,
LoadStyle) are still in main_zig.zig because the matching
blocks/asset_wiring.zig + blocks/resource_loader.zig extractions
(plan steps 4-5) haven't landed yet; the new submodule reaches them
through @import("../../main_zig.zig") in the meantime. When those
blocks land, the import flips to the new module paths and the pub
markers in main_zig.zig revert to private.

Refs: #183
Builds on: #202 (umbrella)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
apotema added a commit that referenced this pull request May 25, 2026
… of #183)

Per the plan in #193. Moved buildSetupCode (raylib/sdl/bgfx/wgpu
loop-backend setup) + buildGuiDrawCode to
src/codegen/lifecycle/loop.zig (235 lines). Re-exported from
main_zig.zig so the orchestrator's references stay unchanged.

HIGH-risk extraction per the plan -- preserves call order across
writeXxxWiring helpers + emitResourceLoad. Verified bit-identical
across all 8 working bundled examples (scripts/gen_all_examples.sh);
sokol_imgui still soft-fails because the labelle-imgui sibling repo
is absent, same as on origin/main.

On this branch the helpers reached by buildSetupCode
(writeImageBackendWiring, writeAudioBackendWiring,
writeFontBackendWiring, emitResourceLoad, LoadStyle) are still inline
in main_zig.zig -- pubified here so the new module can reach them.
When the parallel blocks/asset_wiring + blocks/resource_loader cuts
(PRs #197, #199) land, lifecycle/loop.zig's imports flip to the new
modules and the pub markers in main_zig.zig retire alongside the
moved bodies.

main_zig.zig shrank 3817 -> 3628 lines (-189).
zig build test: 506/510 passed (4 skipped) -- unchanged from baseline.

Refs: #183
Builds on: #202 (umbrella)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@apotema

apotema commented May 25, 2026

Copy link
Copy Markdown
Contributor Author

Closing — content is fully on the umbrella branch feat/183-split-main-zig (via the 12-PR stack that merged into it). Review the complete refactor on #202.

The branch refactor/183-main-zig-investigation (and refactor/183-extract-next for #195) is left in place — git history is preserved.

@apotema apotema closed this May 25, 2026
apotema added a commit that referenced this pull request May 25, 2026
Three findings from gemini on #193, all pre-existing bugs the PoC
extraction surfaced. Fixed only in src/codegen/scan.zig — main_zig.zig
already re-exports these symbols (no duplicate copies to keep in sync).

- :191 — errdefer per dupe before append, prevents leak when one
  step in the chain fails.
- :504 — same shape around extractConstructsString + 3 dupes; the
  optional constructs allocation gets its own errdefer guard. Also
  applied to the parallel pin_styles_out and coercions_out appends
  in the same loop for consistency.
- :757 — std.debug.print -> local stderrPrint helper (matches the
  writeStderr pattern in main.zig / cache_cmd.zig; std.log.err can't
  be used because it trips the test-runner log interceptor, same
  rationale documented in src/scene_manifest.zig).

Bit-identical generated output across all bundled examples (8/8 of
the working ones — sokol_imgui was already failing pre-fix and is
unrelated). zig build test green: 506/510 (4 skipped), same as
baseline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant