Add test endpoint for warm test re-runs without restart - #109
Merged
Conversation
Measured on Mendix 11.13.0. Registering one Java custom request handler at boot gives the test runner an entry point it can invoke repeatedly, instead of triggering tests from the after-startup microflow (a boot hook, so every re-run needs a full runtime restart). Core.getMicroflowNames() lets the handler resolve microflows by name at request time, so the Java is written once and never regenerated as tests change. Measured: unchanged suite re-run 30.55s -> 0.084s; edit-then-re-run 30.55s -> 4.29s (almost entirely the existing --watch rebuild). The load-bearing finding is that the handler survives reload_model: the runtime JVM PID is unchanged, after-startup does not re-run, and the already-registered handler resolves the new model's microflows. Verified in one reload with an edited test and a test created after boot. Also records what is not settled: the endpoint is unauthenticated and executes arbitrary microflows under a system context, and there is no Core.removeRequestHandler. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ
'mxcli test --local' triggered tests from the project's after-startup microflow. That is a boot hook, so a re-run needs a full runtime restart, a failing test manifests as a failed boot, and results have to be scraped out of the runtime log. Local runs now go through the request handler spiked in 2952c2a: startup registers an HTTP endpoint and runs nothing, each test is its own MxTest.Test_<id> microflow invoked by name, and the verdict comes back in the response. A throwing test therefore fails only itself, and because each test has its own variable scope the suffix-renaming the monolithic runner needed is gone. The endpoint executes microflows under a system context, so it is gated four ways, each verified against a live 11.13.0 runtime: - no MXCLI_TEST_TOKEN in the environment => the handler is not registered at all, so a project that kept the MxTest module through a failed cleanup exposes nothing when deployed elsewhere - missing/wrong X-MxTest-Token => 401, compared in constant time - non-loopback caller => 403 - an mf outside MxTest.Test_* => 403 The token is minted per run and reaches the runtime through its environment (new LocalRuntimeOptions.Env), never written into the project. Probing the live runtime also turned up an unfiltered /list returning every microflow in the app; it is now clamped to the test namespace. Docker keeps the after-startup mechanism, which needs no secret and no loopback assumption; --legacy-runner selects it for a local run too. Cleanup additionally removes the generated javasource/mxtest tree, which DROP JAVA ACTION leaves behind. Each gate test was verified to fail against a stubbed guard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ
'mxcli test -p app.mpr --local' failed with MxBuild's "the project file path should be an absolute path" and a page of Windows sample requests, which tells a user who typed a relative -p nothing about what to do. 'mxcli run' already handled this (findings #17), but the fix lived in cmd_run.go rather than in the code that talks to MxBuild, so the next caller — StartLocalApp, via the test runner — re-hit the same error. Absolutizing in ServeServer.Build covers every caller, present and future. LocalAppOptions.applyDefaults and testrunner.Run resolve the path too, so DeployDir, the runtime log path, and the paths named in error messages are not derived from a relative value. Tested by pointing a ServeServer at an httptest fake and asserting on the request body actually sent — something the CLI-layer fix could not be tested for, which is part of why it did not generalise. Both new tests were verified to fail against the reverted fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ
'mxcli test --local' booted the app once per invocation, so every re-run
paid the ~30s cold boot even for a one-character edit. --watch keeps the
runtime and the mxbuild serve server up and re-runs the suite on every
change, to a test file or to the project's model.
Measured on an 11.13.0 app: first run ~30s, then ~2.0s from editing a test
to a verdict on screen and ~2.1s from editing the microflow under test.
Every re-run in the session applied via reload, not restart.
Two hazards this loop has that the run --local dev loop does not:
- The runner writes to the project it is watching, so injecting the test
microflows moves the very mtime being polled. Baselines are taken after
the injection and rebuild settle; getting this wrong is an infinite
rebuild loop, verified absent by idling a session.
- The injected set changes during the session, so cleanup drops what is
currently injected rather than what was injected at boot — otherwise a
test added mid-session is left behind in the user's project. A deleted
test's microflow is dropped explicitly, since CREATE OR REPLACE says
nothing about removal and a lingering flow would keep reporting a stale
pass.
Ctrl-C stops the runtime, restores the project, and says so. --watch is
rejected with --legacy-runner, without --local, and with --skip-build:
those paths can only re-run by restarting.
Verified live across a session that edited, deleted and added tests and
changed the microflow under test, then confirmed the project was fully
restored. Each new test was checked to fail against a stubbed guard.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ
'mxcli test --local --watch' keeps a runtime warm within a session, but a session still opens with a ~30s cold boot. --attach removes even that by running against an app already up: 2.83s for the first attached run and 2.30s for a repeat, with the dev app still serving throughout. Attaching cannot be unilateral, and the reason shapes the design. The endpoint's handler is registered by the after-startup microflow, which runs only at boot, and its token comes from the runtime's environment — neither can be added to an app that is already running. So the dev loop opts in with 'mxcli run --local --test-endpoint', which is also where the decision belongs: hosting the endpoint means the developer's own app carries it, and tests write to the database they are looking at (said on every attach, not just in the docs). What a second process can drive is the dev loop's serve and admin APIs, both plain loopback HTTP. So an attach applies its own injections deterministically rather than waiting on someone else's watcher, and does not require --watch. The host publishes .mxcli/test-endpoint.json (0600, written-then-renamed) with the ports, token, admin password and its PID; the PID check turns a handshake left behind by a SIGKILLed dev loop into one clear message instead of a confusing connection error later. The project's own after-startup microflow is chained rather than displaced, so the app still boots normally, and the endpoint is removed on exit. Ownership is strict: an attach adds and removes only its own test microflows. The endpoint, the after-startup setting and the MxTest module belong to the hosting loop. A change needing a runtime restart is refused rather than half-applied. Running it live caught a bug review had not: the M2EE admin API and the test endpoint are different secrets, and passing the endpoint token to the admin API failed with "Authentication failed" after the test microflows had already been injected. The handshake now carries the admin password separately, taken from the resolved value so an override still works. Help, syntax topics, skills and site docs updated throughout; the running-tests page also no longer claims Docker is required. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ
Brings the branch up to date with main (PRs #107, #108 and the commits behind them) so PR #109 merges cleanly and CI runs against the current base. No conflicts. The fix-issue.md symptom table merged via the union driver as intended — both this branch's rows and main's survive, with no duplicates. Full suite green on the merged tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements a token-guarded HTTP endpoint for running tests against a local Mendix runtime without restarting. This replaces the after-startup microflow mechanism for
mxcli test --local, reducing re-run time from ~30s to ~0.08s.Key Changes
New test endpoint mechanism (
endpoint.go,client.go,generator_endpoint.go):Watch mode (
watch.go,watch_test.go):--watchflag keeps runtime and build server up across runs--legacy-runnerand Docker path (both require restart)Attach mode (
runner_attach.go,handshake.go,handshake_test.go):--attachflag connects to an already-running app hosting the endpointtest-endpoint.json) carries live credentials, written 0600, cleaned up on exitHosted endpoint (
host.go):mxcli run --local --test-endpointinstalls the endpoint into the projectPer-test microflows (
generator_endpoint.go):New options in
RunOptions:LegacyRunner: forces after-startup mechanism on local runs (escape hatch)Watch: re-run on file/model changesAttach: connect to existing endpointDocumentation:
docs/15-testing/SPIKE_test_endpoint_request_handler.mdl)Implementation Details
mxcli run --local --watchdev loops; tests can attach and re-run without stopping the app.--legacy-runnerflag preserves old after-startup behavior for cases where endpoint misbehaves.Measurements
https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ