Skip to content

fix(pages): read a pluggable widget's microflow datasource from MicroflowSettings - #57

Closed
ako wants to merge 8 commits into
mainfrom
claude/mxcli-issues-analysis-g3qmt6
Closed

fix(pages): read a pluggable widget's microflow datasource from MicroflowSettings#57
ako wants to merge 8 commits into
mainfrom
claude/mxcli-issues-analysis-g3qmt6

Conversation

@ako

@ako ako commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Fixes mendixlabs/mxcli#795 — DESCRIBE PAGE drops a datagrid's microflow DataSource, so round-tripped MDL recreates the grid with no data source.

(The issue number refers to mendixlabs/mxcli, not this fork.)

The bug

A DataGrid2 bound to a microflow described as:

datagrid g1 {
  column "Key" (Attribute: Key, Caption: 'Key')
}

— no DataSource. A database from grid on the same page described correctly, which is what made the loss look selective.

The reporter was right that the model stores it correctly. The page BSON holds:

"DataSource": {
  "$Type": "Forms$MicroflowSource",
  "MicroflowSettings": {
    "$Type": "Forms$MicroflowSettings",
    "Microflow": "MyModule.DBG_ListBucketObjects"
  }
}

The name lives in the nested Forms$MicroflowSettings; there is no top-level Microflow key. extractDataGrid2DataSource looked up ds["Microflow"], got "", and returned nil. The describe formatter's case "microflow" branch was correct all along — so neither the write path nor the stored model was ever wrong. Purely a read bug.

The fix

Four readers each had their own copy of this lookup and disagreed with each other — two read the nested settings, one read the top-level key, one did both. They now share microflowSourceRef / nanoflowSourceRef, which read the nested settings and fall back to the legacy top-level key so older files still round-trip.

Two further gaps the regression tests turned up while covering this:

  • A Gallery whose source used the legacy flat shape was unreadable too. Gallery delegates to parseCustomWidgetDataSource, which handled only the nested form, and the flat fallback in its own switch is not reached for a CustomWidget gallery.
  • Forms$NanoflowSource was missing entirely from the DataGrid2 and Gallery switches, so a Studio-Pro-authored nanoflow datasource described as nothing.

Left alone deliberately: CustomWidgets$CustomWidgetNanoflowSource (a different metamodel type whose Nanoflow key really is top-level) and the Forms$MicroflowAction reads (actions, not datasources).

Also corrects the datasource table in the create-page skill, which documented datasource: microflow Module.GetData() — the trailing () does not parse, as the reporter noted.

Verification

Reproduced on main before the change, against a real project:

before after
datagrid + microflow datagrid g1 { datagrid g1 (DataSource: microflow …)
datagrid + database correct correct (unchanged)
gallery + microflow correct correct (unchanged)

The round-trip the issue is actually about — describeexecdescribe — now produces byte-identical output.

make build, make test, make lint and make check-mdl all pass.

New coverage in mdl/executor/cmd_pages_describe_flowsource_test.go: a table over both extractors covering the nested shape, the legacy flat shape, and nanoflow, plus direct tests of the shared helpers including non-map MicroflowSettings (no panic). Verified these fail against the pre-fix code — including the two extra gaps, which I had not predicted from reading alone.

Repro fixture: mdl-examples/bug-tests/795-datagrid-microflow-datasource-describe.mdl, and a row added to the .claude/skills/fix-issue.md symptom table.

Out of scope, found while testing

Two things I did not touch, worth their own issues:

  1. A nanoflow datasource on a pluggable datagrid is dropped at write time. mxcli exec logs modelsdk: pluggable widget data source *pages.NanoflowSource not yet supported — rerun with MXCLI_ENGINE=legacy and then still prints Created page. The read fix here covers Studio-Pro-authored nanoflow sources; mxcli still cannot author one, and it fails quietly.
  2. The -- Context: comment names the microflow as the context entity$currentObject (MyModule.DBG_ListBucketObjects) instead of the entity the microflow returns. Affects datagrid and gallery alike; cosmetic, describe-output only.

🤖 Generated with Claude Code

https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA


Generated by Claude Code

claude added 8 commits July 30, 2026 14:29
…flowSettings (mendixlabs#795)

DESCRIBE PAGE emitted `datagrid g1 {` with no DataSource for a DataGrid2
bound to a microflow, so re-applying the describe output recreated the grid
with no data source at all. A `database from` source described correctly,
which made the loss look selective.

A Forms$MicroflowSource stores the microflow name in the nested
Forms$MicroflowSettings ("MicroflowSettings" -> "Microflow") — what the write
path emits and what Studio Pro stores. extractDataGrid2DataSource looked up a
top-level "Microflow" key, got "", and returned no datasource. The describe
formatter's `case "microflow"` branch was correct all along, so nothing was
ever wrong with the stored model or the write path: purely a read bug.

Four readers had their own copy of this lookup and disagreed — two read the
nested settings, one read the top-level key, one did both. They now share
microflowSourceRef / nanoflowSourceRef, which read the nested settings and
fall back to the legacy top-level key so older files still round-trip.

Two gaps the regression tests found while covering this:

- a Gallery whose source used the legacy flat shape was unreadable too: the
  correct reader it delegates to handles only the nested form, and the flat
  fallback in its own switch is not reached for a CustomWidget gallery
- Forms$NanoflowSource was missing entirely from the DataGrid2 and Gallery
  switches, so a Studio-Pro-authored nanoflow datasource described as nothing

Left alone deliberately: CustomWidgets$CustomWidgetNanoflowSource, a different
metamodel type whose Nanoflow key really is top-level, and the
Forms$MicroflowAction reads, which are actions rather than datasources.

Also corrects the datasource table in the create-page skill: it documented
`datasource: microflow Module.GetData()`, and the trailing `()` does not parse.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA
A construct the codec engine cannot represent — a nanoflow datasource on a
DataGrid2 is the reachable case — was logged and discarded while `mxcli exec`
still printed `Created page`. The grid landed with no data source at all.

The converter does return an error; it just had nowhere to go. Child content
is serialized through widgetobj.ChildSerializer, whose methods return BSON
with no error channel — the interface change flagged by the TODO in
mdl/backend/widgetobj/builder.go. Until that lands, the failure is recorded
in the modelsdk backend and drained by the page/snippet write entry points,
so the statement fails with the actionable message instead of the write
succeeding with data missing. ADR-0004 already required this: where the codec
path cannot reproduce a construct the backend refuses the op rather than
dropping it.

Authoring a pluggable nanoflow datasource still needs MXCLI_ENGINE=legacy,
which writes it correctly — verified, since the error message says so.

Also adds the missing `nanoflow` case to the DataGrid2 and Gallery describe
formatters. Reading such a source works after mendixlabs#795, but the formatter had
only database/microflow/parameter branches, so a legacy-written or
Studio-Pro-authored nanoflow grid still described with no DataSource.

Any new write entry point that builds pluggable widgets must drain too; the
symptom table records that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA
…urns

A data container bound to a microflow or nanoflow reported the *flow* as its
context entity:

  datagrid g1 (DataSource: microflow Module.GetOrders) {
    -- Context: $currentObject (Module.GetOrders)   <- the flow, not the entity

The datasource reference was used verbatim. That is correct for a database
source, where the reference is the entity, and wrong for a flow source, where
it is the flow's own qualified name. Besides the misleading comment, the
context is passed down to the column and child-widget readers, which resolve
attributes against it.

The flow's return type is resolved instead, taking the entity from an Object
or List return. When the flow cannot be resolved, or returns a scalar, the
reference is kept — the previous behaviour, so nothing gets worse.

Resolution goes through ListMicroflows/ListNanoflows plus the container
hierarchy rather than GetRawUnitByName, which the flow builder's fast path
uses: that method is unimplemented on the modelsdk engine, which is the
default, so the list path is the one that actually resolves.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA
)

`alter entity X set allow_create_change_locally = true` reported success and
left the flag off. So did every other remote setting, and `describe external
entity` rejected the entity outright with "is not an external entity
(source: )".

The write path was never the problem: externalEntitySourceToGen switches on
e.Source and handles all three Rest$OData* flavours. The *read* filled none of
them — entityFromGen recognised only DomainModels$OqlViewEntitySource, so an
external entity came back with an empty Source and every remote field zeroed.
Setting a flag then wrote it onto a model that no longer knew the entity was
external, and the update rebuilt the entity without its source at all.

The legacy engine parses all three flavours (sdk/mpr/parser_domainmodel.go),
so this was a modelsdk-engine gap — and modelsdk is the default. Under
`--engine legacy` the same commands worked, which is a useful bisect signal.

Now mirrored for:

- Rest$ODataRemoteEntitySource — service, entity set, remote name, the
  capability flags, CreateChangeLocally and the remote key
- Rest$ODataEntityTypeSource — service, type name, IsOpen and the remote key
- Rest$ODataPrimitiveCollectionEntitySource — service

Updatable is deliberately left zero: the storage type has no such field
(updatability is per attribute via Rest$ODataMappedValue), there is no gen
accessor, and the writer does not emit one, so read and write stay symmetric.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA
…y re-import

`create or modify external entities` reset "Allow creating and changing
objects locally" to false on every run, so the setting survived only until the
next import — the other half of mendixlabs#782.

applyExternalEntityFields stamps every field on both the create and the update
path. That is right for the capability flags (Countable / Creatable /
Deletable / SkipSupported / TopSupported), which are derived from the service's
capability annotations and should be refreshed from the metadata. It is wrong
for CreateChangeLocally, which no OData contract describes: it is a local
modelling choice, so the top-level branch now leaves it alone. A newly imported
entity arrives zero-valued, which is Mendix's default.

The entity-type branch still clears it, since that storage type has no such
field — a re-import that reclassifies an entity as a derived type would
otherwise leave a stale value in the model.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA
…endixlabs#803/mendixlabs#804)

Three defects in one teardown, which together left a tested project
permanently mutated.

mendixlabs#803 — the after-startup setting was never restored. getAfterStartup trimmed
quotes before trailing punctuation, and DESCRIBE SETTINGS separates properties
with commas, so

    AfterStartupMicroflow = 'MyFirstModule.ASU_Startup',

parsed to `MyFirstModule.ASU_Startup',` and the restore statement became

    ALTER SETTINGS MODEL AfterStartupMicroflow = 'MyFirstModule.ASU_Startup','

which does not parse. The failure printed as a warning, so the run looked
clean while the project was left pointing at MxTest.TestRunner — the microflow
cleanup then deleted. Trailing `,;` now comes off before the quotes, the value
is re-emitted through a helper that doubles embedded quotes (Mendix's escape,
never backslashes), and cleanup returns its errors instead of printing them:
a failed restore now fails the run and names what was left changed.

mendixlabs#804 — cleanup dropped MxTest.TestRunner but not the MxTest module it created,
so an empty module accumulated on every run. It now drops the module — but
only when the run created it. CREATE MODULE is idempotent, so a project that
already had its own MxTest module would otherwise have had it deleted; that
case still removes just the generated microflow. Cleanup also re-checks that
the module is present, so a run that failed before injection landed does not
report a spurious cleanup failure.

mendixlabs#802 — the Security Level was forced OFF before building and restored to a
hardcoded PRODUCTION, silently changing projects that legitimately run at
another level, and breaking any project whose published REST/OData services
use custom authentication ("App security is off, but custom authentication is
enabled for this service"). The after-startup microflow runs in an
administrative context and is not subject to the Security Level, so this
bought nothing. It is gone; the setting is the project's business.

What cleanup must restore is now captured in a projectState before the first
mutation, and the setup/cleanup statements are built by pure functions, so the
restore is testable without a project or Docker.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA
…endixlabs#791)

A loop containing a split whose branch does `continue` wrote successfully and
could not be opened in Studio Pro:

    System.Collections.Generic.KeyNotFoundException: The given key
    '806fca46-5c4b-46f8-a890-4d24dd29c24f' was not present in the dictionary.

microflowObjectToGen had no case for Microflows$BreakEvent or
Microflows$ContinueEvent, and its default branch returns nil. The event object
was therefore dropped at serialization while the SequenceFlow pointing at it
was written, leaving a DestinationPointer to a GUID that exists nowhere in the
document — the key Studio Pro cannot resolve. Confirmed by dumping the written
microflow and checking every *Pointer against the object IDs: one dangling
pointer, and no ContinueEvent anywhere. The legacy engine serializes both, so
this only affected the default engine.

The identical bug had already been found and fixed for Microflows$ErrorEvent,
whose case still carries the comment explaining it. Break and continue were
missed.

MDL051 goes with it. That rule rejected a conditional `break` at check time
and pointed users at a guard-variable workaround, explicitly "until the
serialization is fixed" — it now is, and keeping the rule would reject valid
MDL. Note it only ever covered `break`: `continue` had no guard, which is how
this reached a user as a corrupt project. Its negative fixture becomes a
positive one, and its test now asserts the pattern is accepted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA
…ies (mendixlabs#790)

A loop box was drawn far wider than its contents — 880px around three
activities spanning 440px, leaving ~290px of empty area inside the frame.

measureStatements sums each element's full width and adds HorizontalSpacing
between them. But HorizontalSpacing is a centre-to-centre pitch: the builder
centres each activity on posX and advances by exactly that, so counting it on
top of each width over-measures a run of n simple activities by
(n-1)*ActivityWidth — 240px for three. The loop width is derived from that
measure, so the box inherited the error.

Loop sizing now uses measureStatementsSpan, which returns the true extent
(n-1)*HorizontalSpacing + ActivityWidth for a run of simple activities. The
example above goes from 880 to 640 (the body occupies 90..530 inside it), and
nested loops shrink at both levels.

A compound element (IF/split, nested loop) advances posX by merge geometry
this function cannot reproduce without duplicating the builder. An earlier
attempt to approximate it under-sized the box and pushed the trailing activity
30px outside the frame — worse than a box that is merely too wide — so those
runs fall back to the conservative measure and are unchanged. Verified by
checking that every child of a LoopedActivity lies within its box, across
loop-with-if, nested-loop, single-statement and multi-statement bodies.

Not addressed: the other half of mendixlabs#790, "position annotations sometimes
change". Explicit @position annotations round-trip exactly, and three
successive describe/exec cycles produce byte-identical output, so there is
nothing reproducible to fix here — the reporter also noted it was
inconsistent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA

ako commented Jul 30, 2026

Copy link
Copy Markdown
Owner Author

Superseded — split into four topic PRs, one per area, so each can be reviewed and taken upstream on its own:

All 8 commits and every changed file are accounted for across the four branches — verified by diffing the file sets, with nothing added or dropped. Each branch passes make build, make test, make lint and make check-mdl independently off current main.

Closing this one in favour of those.


Generated by Claude Code

@ako ako closed this Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants