Server-side backend: a native, JVM-free runtime for Codename One handlers - #5741
Server-side backend: a native, JVM-free runtime for Codename One handlers#5741shai-almog wants to merge 140 commits into
Conversation
The clean (non-Objective-C) target could translate a Java main() and run it, but
not much more: main(String[]) was handed JAVA_NULL, so a translated program could
not read its own command line, and there was no way to read the environment, open
a file or read stdin. Every knob had to be a compile-time macro, which is why the
GC benchmarks are parameterised the way they are.
- argv reaches main(String[]) via cn1MainArgs, skipping argv[0] the way Java does
- System.getenv(String)
- java.io.FileInputStream / FileOutputStream over C stdio, so the same code
serves the Windows target, which has no unistd.h
- java.io.StandardInputStream behind System.in. Not a FileInputStream: stdin is
not seekable, so skip and available cannot be answered by seeking
Separately, CHECKCAST. BC_CHECKCAST expanded to nothing, so a failed cast handed
the wrong object to the next instruction and the target type's fields were read
out of it -- a native crash no Java catch can see (issue #5531). Implementing the
macro alone would have changed nothing: BytecodeMethod DELETES the CHECKCAST
instruction before codegen ("gets in the way of other optimizations"), so nothing
ever reached TypeInstruction. Array stores had the companion hole -- AASTORE was
bounds-checked but never covariance-checked, and the macro's own comment claimed
otherwise.
Both are now enforced under -Dcn1.checkedCasts=true, which also drives retention
of ClassCastException and ArrayStoreException so the emission and the classes can
never disagree and leave an unresolved symbol. Opt-in, because turning it on
changes the outcome of app builds that succeed today; a server-side build parsing
untrusted input should always enable it.
Verified against vm/tests: 80 tests, no regressions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The next stage is a standalone server rather than a Lambda, and the first
question it asks is whether a connection can have a thread. That needed a number,
so ThreadCost parks N threads and holds them while RSS is read from outside.
Measured with 512 parked threads:
musl/arm64 (the deployment target) 243 KB/thread
macOS/arm64 118 KB/thread
Attribution on Linux, by ablation:
callStack arrays (1024 -> 128) -50 KB
pendingHeapAllocations (4096 -> 256) -27 KB
try blocks (500 -> 32) -15 KB
shadow stack (16536 -> 2048) 0 KB
thread stack (16MB -> 256KB) 0 KB
Two of those are worth recording because they are the opposite of what the
macOS numbers suggested. The shadow stack, the biggest single allocation at
258KB, costs nothing resident on Linux -- shrinking it changes the number not at
all, though on macOS it looked like the dominant cost. And the pinned 16MB thread
stack is free: it is reserved, never committed.
The five sizes are now #ifndef-guarded so an A/B can override them with -D. They
were unconditional #defines, so a -D was silently ignored -- the redefinition
warning is suppressed by the generated code's -w, which is how the first round of
ablations produced three identical numbers and no conclusion.
The shadow stack is now mapped rather than malloc'd and memset in full. That is a
spawn-path win (258KB of stores per thread creation), not a footprint win; the
comment says so rather than implying the measurement it did not produce.
The conclusion for the server design: at 155-243 KB even with every buffer
shrunk, ten thousand connections is 1.5-2.4GB of threads. A connection cannot have
one. The design is a reactor with a bounded worker pool, where a few dozen threads
cost a few megabytes and the connection is just an fd.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
throwException walked the try-block stack looking for a handler and, when it found none, RETURNED. The generated code then carried on with the statement after the throw, with the method's locals in whatever state the failed operation left them. On an app target something upstream nearly always catches -- the EDT's own try -- so this stayed invisible; a server binary has nothing above main. What it looked like in practice: a database client whose TLS handshake was rejected threw, Database.open "returned" a null, and the program segfaulted two statements later on the null. The message that would have named the real cause was never printed, and a program that threw out of main exited with status 0. The clean target now prints the exception, its message and a stack trace, and exits 1. Every other target keeps today's behaviour: making this fatal everywhere would change what apps that ship today do, so the generated main() opts in and nothing else does. Two details the fix needed. The message is fetched separately because the pre-rendered stack string carries only the type, and on a server the message is the actionable half. And the try depth is reset to zero before rendering: the search leaves it at -1, and a Java method that saves and restores a negative depth corrupts what it restores into, which turned the reporter itself into a SIGBUS. Also here, because the same audit found it: java.lang.System.in is a static field, so every translated program reaches StandardInputStream's natives, and the JavaScript backend had no category for them -- which turned the core-slice completeness gate red for code that never touches stdin. They are marked unsupported there, as java.io.File already is: a browser has no process stdin. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both are one-line consequences of the same C rule, found by building the same program two ways. ATOMIC_VAR_INIT on an atomic POINTER is rejected by clang 14 -- which is what Debian bookworm ships, and therefore what the glibc backend builder image uses -- as "initializer element is not a compile-time constant". The generator emits it for every `volatile` static reference field, so any such field in ordinary user code failed to build there. A static object is zero-initialized by the language, so the initializer is dropped; the macro is deprecated in C17 and gone in C23 regardless. CN1_RESUME_THREAD referenced gcParkCaptured unconditionally, but that field only exists when conservative roots are compiled in. So -DCN1_DISABLE_CONSERVATIVE_GC_ROOTS -- the A/B arm vm/CLAUDE.md documents -- did not build at all, and the one measurement that isolates the conservative scan's cost could not be taken. It is now behind a macro that compiles away with the field. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A virtual thread runs Java on a stack of its own, so parking one is a stack
switch of a couple of nanoseconds rather than a blocked OS thread. Measured
round trip on arm64: 2.1ns.
The runtime is three files -- cn1_virtual_thread.{h,c} and the context switch,
which has to be assembly because glibc aborts a cross-stack longjmp under
_FORTIFY_SOURCE and musl has no makecontext. aarch64 and x86_64 are implemented;
anywhere else the header's stubs answer "there is no virtual thread here", which
is the truth, and every caller folds away at compile time.
The collector had to learn about them, because a virtual thread breaks two of its
assumptions silently:
- A carrier RUNNING a virtual thread has its stack pointer inside that virtual
stack, so the [sp, base) bounds test rejected it and skipped every
conservative root the thread held.
- A PARKED virtual thread is referenced by nothing the collector walks, while
its stack still holds Java references in C temporaries.
Both are served from a registry snapshot taken once per cycle before any thread
is stopped: walking the live registry would take its mutex, and a thread frozen
by the stop signal may be the one holding it.
Also here, because they are what made the above work: the translator emits the
runtime into every generated project, and CN1_RESUME_THREAD yields a virtual
thread rather than sleeping the carrier it runs on -- a carrier hosts many
virtual threads, so sleeping it freezes all of them.
Carried along in the same change: LinkedHashMap runs its eviction hook only on a
real insertion, as java.util does, which also drops an allocation per insertion;
a generated mapper can serialise straight to JSON instead of filling a map and
walking it back, measured 2.05x/1.51x/2.81x on a four-property object with output
asserted byte-identical; and a repeated CHECKCAST is dropped when it immediately
follows the identical one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CN1_RESUME_THREAD waited out a collection with usleep(1000). Two things make that expensive on the backend and neither is visible at the call site. It sleeps the CARRIER, and a carrier hosts many virtual threads: hostCount is min(workers, cores), so on a two-core pin sixty four connections share two carriers. One carrier sleeping a millisecond freezes about thirty two connections that were ready to run, which is the shape of a server whose median is healthy and whose tail is not. And it is a sleep-poll, so the wait is quantised to the sleep interval however briefly the flag was actually held. The measured worst case was 1923us: two iterations of a 1ms sleep waiting for something that had long since cleared. The pacing park already yielded here; this site did not, and it is the hottest of the four -- once per syscall return, 204105 times in a twenty second run against 9 for the handshake. Platform threads still sleep, having nothing to yield to, and off the backend the stub answers "not virtual" so the macro folds back to exactly the old loop. This shortens the wait; it does not remove it. The thread is still held until the collector has drained the whole worklist reachable from its roots rather than merely captured them, which is a separate question and a larger one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cn1SpawnVirtualThread and cn1CreateThreadLocalData were declared inside #ifdef CN1_CONSERVATIVE_GC_ROOTS. Neither has anything to do with how the collector finds its roots, and burying them there broke -DCN1_DISABLE_CONSERVATIVE_GC_ROOTS -- the precise threadObjectStack arm that vm/CLAUDE.md documents -- with an undeclared cn1SpawnVirtualThread in the backend's native sources. C being what it is, the implicit declaration then also produced an int-to-pointer conversion, so the failure named the wrong thing. Found while measuring that arm rather than by building it, which is the point: nothing builds it. The default build is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CN1_RESUME_THREAD is a safepoint: it can park the thread on a timed wait while a collection runs, and that overwrites errno. Reading errno after it recorded the WAIT's outcome rather than the read's, so lastError handed Java an error belonging to something else entirely. Captured at the syscall instead. The do/while EINTR retry idiom elsewhere is already safe -- it reads errno before the resume. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mark phase signals every thread and spins until it answers, so it can scan the thread's native stack conservatively. A thread that never answers is not scanned either way -- the caller returns 0 and reads nothing -- so the wait buys literally nothing, and one such thread cost 267ms of a 280ms mark, every cycle. Count consecutive timeouts per thread and skip a thread that has failed three of them, re-probing every 64th attempt so one that becomes responsive is picked back up, and clearing the count the moment it answers. The forced-stop escalation (issue #5537) must NOT be throttled this way, so the implementation takes a maySkip flag and the escalation passes 0. It retries every CN1_GC_SAFEPOINT_WAIT_MAX_US precisely to ride out a transient or descheduled handler; skipping those retries would leave the collector waiting on threadActive for tens of seconds, turning a recoverable timeout into exactly the whole-VM pause the escalation exists to prevent. Measured on the server workload: stackMs 269 -> 0.20. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sembly Two halves of one bug. Virtual threads were gated on a build flag that only the server build set, and the flag was justified by an Xcode misfiling it was working around: Xcode has no mapping for the .S extension, so an unrecognised one becomes `lastKnownFileType = file` and lands the file in the RESOURCES phase, where it is copied into the bundle and never assembled. The iOS target then failed to link naming _cn1VirtualThreadSwitch, whose source was sitting right there in the project. Gating the feature off made the misfiled resource inert, so the phone target linked and the misfiling stayed hidden. Fix the misfiling instead: .S maps to sourcecode.asm.asm (preprocessed, which the capability gate in the file needs) and .s to sourcecode.asm, and both route into the Sources phase rather than Resources. Every future assembly file gets this too. That removes the reason for the flag, so the gate becomes a capability test: on anywhere the switch is written for -- aarch64 and x86_64, excluding Windows, whose calling convention needs its own prologue -- virtual threads are on. There is no separate "server build" of the VM; a flag would only mean the feature is off in every build nobody remembered to set it in. Elsewhere the header's no-op stubs answer "there is no virtual thread here", which is true, so the collector needs no #ifdefs and every call folds away. CN1_DISABLE_VIRTUAL_THREADS forces that path. The predicate is repeated verbatim in the .S, which is preprocessed assembly and cannot include the header -- the two must stay identical or the link breaks on the switch symbol. Also excludes LinkedHashMap from the copyright gate: it is Apache Harmony source and keeps its Apache-2.0 notice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Turning virtual threads on by capability rather than by a flag nobody set made
three latent bugs reachable at once, all the same shape: the context switch was
copied into the generated project and never assembled, so the C half linked
against a symbol whose source was sitting in the same directory.
- CMake globbed *.S only for the LINUX app type, and only when embedding
resources -- the condition belonged to the resource blob, which used to be
the only .S there is. Now any .S present drives both the ASM language and the
glob, on every cmake target.
- The WINDOWS app type is also cross-built with clang on a POSIX host, where
_WIN32 is undefined, the switch is live, and MSVC's inability to assemble GNU
syntax is irrelevant. That is a question about the compiler, and CMake can
only answer it after project() has enabled C, so it is asked there rather
than guessed from the app type. Under MSVC the variable stays unset and
expands to nothing.
- Xcode has no mapping for .S at all, so it became `lastKnownFileType = file`
and landed in the RESOURCES phase, shipped into the bundle and never built.
sourcecode.asm is the identifier for both spellings: Xcode's own
StandardFileTypes.xcspec lists it as `Extensions = (s)` with
`GccDialectName = assembler-with-cpp`, which is the preprocessing the file's
capability gate needs. The neighbouring sourcecode.asm.asm is for .asm.
Tests. BackendUncaughtExceptionTest needed a support class that does not exist
here, and only ever reached the fix through a server binary; replaced by
UncaughtExceptionIntegrationTest, which builds a clean-target program directly
and asserts the whole contract -- message, stack frame, non-zero exit, and that
execution stops AT the throw rather than carrying on, which is the half the other
three can all pass without.
test_virtual_thread.c was built by nothing. A hand-written context switch with no
enforced coverage could break in any commit and stay green, so
VirtualThreadRuntimeTest drives it from the suite, compiled out of the SAME
staged classpath resources a generated project receives -- which also asserts
those three files are present and agree with each other.
The iOS project test now asserts the assembly is typed as assembly, IS in the
Sources phase and is NOT in Resources. All three: the type alone does not prove
the phase, and the phase alone does not prove it assembles.
The generator's own source set is what caught the last of it. Two copies of
replaceLibraryWithExecutableTarget matched the add_library line by its full
argument LIST -- the shared one in CleanTargetIntegrationTest and a private
duplicate at the bottom of FileClassIntegrationTest. Adding the assembly glob
made both stop matching, so those tests built a library and then failed running
an executable nothing had asked for. The shared one now matches the CALL and
asserts the substitution happened; the duplicate is gone, and FileClassIntegration
uses the shared one like the other twenty-two callers already did.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All five were real. Taken together they are one theme: a virtual thread is a
mutator the collector cannot see by the usual means, and the code that creates
one was doing only half the job.
RUNNING VIRTUAL THREADS LOOKED PARKED. cn1SpawnVirtualThread builds its VM state
with bindToCallingOsThread false, which leaves threadActive FALSE, and nothing
ever raised it. A collection running concurrently therefore treated a mutator
executing Java as parked, and was free to scan or migrate its object stack and
pending-allocation table underneath it -- missed roots at best, corruption at
worst. The flag now moves with the context switch, up on resume and down on
suspend, because a SUSPENDED virtual thread genuinely is parked: the collector
reaches its roots through the registry snapshot instead.
The transition is a weak symbol with a no-op default, not a function pointer.
cn1_virtual_thread.c cannot include cn1_globals.h (the standalone runtime test
builds it with no VM at all), an indirect call on a path whose entire value is
that it costs 2.1ns is not free, and a weak symbol costs a direct call the linker
resolves to the VM's real one when there is a VM.
NOTHING RELEASED THE STATE. cn1VirtualThreadFree knows only about the coroutine.
The VM state spawned beside it holds a 264KB shadow stack, the call-stack arrays,
the pending-allocation table, and one of the NUMBER_OF_SUPPORTED_THREADS slots in
allThreads. A virtual thread per request would have consumed a slot per completed
request and eventually tripped CODENAME_ONE_ASSERT(threadOffset > -1). Added
cn1RetireVirtualThread, which marks the state dead the way an OS thread's death
does and then frees it with the same gcQueuedForDrain deferral the Java finalizer
uses.
THE UNCAUGHT-EXCEPTION EXIT WAS NOT GATED. This is the one that would have
shipped. The generated main() is emitted for every target that has one, iOS and
macOS included, and cn1AbortOnUncaughtException was set unconditionally -- so an
uncaught exception on any thread would have terminated a shipped app. The comment
sitting above it claimed the opposite ("Only this target opts in, so nothing that
ships today changes behaviour"), which was simply false: the enclosing guard is
`if(m.isMain())` and nothing more. Now gated on OUTPUT_TYPE_CLEAN.
BLOCKING STDIN NEVER PARKED THE MUTATOR. System.in.read() waits as long as nobody
types, with the thread left active, so a concurrent collection spun for a
safepoint that could not arrive until a human pressed a key. Bracketed with
CN1_YIELD_THREAD/CN1_RESUME_THREAD like the socket reads -- which then needs the
keep-alive those reads also need, because only an interior pointer into the array
is live across the call and the collector would otherwise sweep the buffer being
filled. Portable here (a volatile store) rather than the Linux port's asm
barrier, because this file also compiles under clang-cl. feof is read before the
resume for the same reason errno is: the resume is a safepoint, and anything
asked afterwards describes the wait.
THE SHADOW STACK WAS FREED THE WRONG WAY. cn1AllocThreadStack falls back to
calloc when mmap is out of MAPPINGS rather than out of memory, and
cn1FreeThreadStack always called munmap. That fails with EINVAL and leaks the
whole stack -- or, on an allocator that returns page-aligned blocks, unmaps
memory the allocator still believes it owns. Which allocator answered is now
recorded and the free is paired to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… does
Mapper.Direct's contract is to produce exactly what
JSONWriter.toJson(toMap(instance)) would. Two fields did not, so a mapper changed
its wire representation on the day it gained a direct writer:
- A null List serialised as `null`, where the map path emits `[]` --
emitFieldToMap builds its ArrayList unconditionally and fills it only when
the source is non-null.
- Enum elements went through toString(). The map path uses Enum.name(), and
deserialisation matches against the declared constants, so an enum that
overrides toString() produced JSON that could not be read back at all.
Every other element kind was checked rather than assumed: appendJsonValue already
maps Date to getTime(), scalars and collections through writeJson, and a mapped
object through its own mapper -- the same three answers emitFieldToMap gives.
Nothing was comparing the two paths, which is why both got through. Every
existing test exercises one route or the other, never one against the other, so
the divergence was invisible to all of them. directJsonMatchesTheMapPathExactly
runs an object with a populated list, an enum list, a Date and scalars, and then
the same class with every list left null, asserting the two routes produce
identical text. It asserts equality of the paths rather than against a literal on
purpose: it keeps holding when a field kind is added, with nobody remembering to
extend a hand-written expectation.
Two things that test needed before it proved anything. It drives the generated
mapper's own toJson rather than Mappers.appendJson, which goes through the
registry -- unpopulated in an isolated classloader, so it fell back to toString()
and compared the map path against "com.example.Swatch@23706db8". And it asserts
the mapper actually implements Mapper.Direct, without which it would compare the
map path with itself and pass while testing nothing. The test enum deliberately
overrides toString() to disagree with name(), so the wrong choice cannot pass.
Also drops a redundant `public` on the interface: PMD's UnnecessaryModifier, and
a zero-findings gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ow needs Two CI breakages, both from this branch making something reachable that had not been reached before. EVERY cn1lib NATIVE CHECK STOPPED AT A MISSING HEADER. cn1_globals.h now includes cn1_virtual_thread.h -- CN1_RESUME_THREAD yields a virtual thread rather than sleeping the carrier it runs on -- and two places stage the port headers into a scratch directory to compile a cn1lib against them. Neither knew about the second file, so both stopped at "'cn1_virtual_thread.h' file not found" before compiling a line: the six ad-cn1lib xcodebuild probes and check-cn1lib-native-sources.py. The workflow's path filters gain the header too, otherwise a future change to it skips the very check that would catch this. java.io.File HAD NO WINDOWS PATH. Its non-ObjC arm is POSIX-only -- unistd.h, dirent.h, access(), X_OK -- and Windows reaches that arm under clang-cl, which is neither __OBJC__ nor POSIX. It went unnoticed because java_io_File_runtime.c is emitted only when an app actually uses java.io.File, and until the clean target became a usable program runtime no Windows build ever did. Now every one of them failed on 'unistd.h' file not found. The Win32 arm: io.h and direct.h for _access, the access-mode constants the MSVC CRT does not define, and FindFirstFile for the directory walk, in the same two-pass shape as the POSIX one (count, allocate, refill) because allocArray can collect and the array must not be built with a find handle open. X_OK maps to an existence check: Win32's access model has no execute bit, and _access REJECTS a mode of 1 rather than answering "not executable". isHidden asks for FILE_ATTRIBUTE_HIDDEN instead of guessing from a leading dot, which means nothing on Windows. Everything else -- stat, remove, rename, mkdir -- the CRT already provides under the same names. Also merges two identical project() branches that SpotBugs flagged as DB_DUPLICATE_BRANCHES: Linux and the clean target answer the assembly question the same way, so they share one branch instead of two spelled alike. The POSIX arm is verified here (FileClassIntegrationTest, 5/5); the Win32 arm can only be verified by CI, which is what reported it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CodeQL java/zipslip, high severity. unzip() built each output path by
concatenating the destination with ZipEntry.getName(), unchecked, so an entry
named "../../x" wrote wherever the archive asked. Both callers unpack a
DOWNLOADED zip -- Groovy for the console, JavaFX for the browser component -- so
the archive is not something the user authored, and the consequence is an
arbitrary file overwritten under their account while they believe they are
unpacking a dependency. CWE-22.
Every entry now has to resolve inside the destination or it is refused. The
comparison is between CANONICAL paths -- resolving the ".." is the whole point --
and it uses java.nio.file.Path.startsWith rather than String.startsWith, for two
reasons. Path compares COMPONENT-wise, so a sibling like "/tmp/dest-evil" is
rejected against "/tmp/dest" where a character-wise prefix accepts it, and giving
the string prefix a trailing separator to fix that then wrongly rejects the
destination directory itself. It is also the shape CodeQL recognises as a
sanitizer: the first attempt here was a correct canonical-path check that the
query still flagged, because a compound `!a && !b` guard did not read as a
barrier.
Two things the fix had to bring with it, both found by writing the test:
- Parent directories are created before extracting. FileOutputStream will not
create them, and a nested entry can arrive before the directory entry that
holds it, so "nested/deep/leaf.txt" in an archive that declares no directory
entries threw FileNotFoundException. That was broken before this change too.
- destDir uses mkdirs rather than mkdir, so a destination more than one level
deep is actually created.
Both streams are closed in a finally, which they were not: an IOException
mid-extract leaked the descriptor.
The test builds the malicious archive rather than checking one in -- a committed
zip that escapes its destination is an awkward thing to keep in a repository, and
building it puts the attack in front of the reader. Verified non-vacuous by
reverting the fix: 2 failures against the old code, 0 against the new.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the unistd.h/dirent.h dependency got clang-cl past the first error and
into four more, all the same kind -- POSIX spellings the MSVC CRT does not have:
- `redefinition of 'timeval'`. <windows.h> pulls in <winsock.h>, whose timeval
collides with the one cn1_win_compat.h defines. WIN32_LEAN_AND_MEAN keeps
winsock out, and nothing here wants it.
- S_ISDIR / S_ISREG undeclared. The CRT has the st_mode BITS but not the macros
that test them, so they are defined from _S_IFMT/_S_IFDIR/_S_IFREG.
- PATH_MAX undeclared -- MAX_PATH is the Win32 spelling.
- realpath undeclared. _fullpath is the equivalent, but it takes
(destination, source), the REVERSE of realpath's (source, destination), so
the macro swaps them. Getting that backwards compiles and canonicalizes the
wrong string in silence. It also resolves a path that does not exist rather
than failing, which is the more useful answer for getCanonicalPath.
The POSIX arm is unchanged and still verified here (FileClassIntegrationTest,
5/5). The Windows arm is verified only by CI, which is what reported both rounds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two Windows-only build breaks in this branch's own new code, both invisible on
the POSIX legs.
`i->gcPthread = 0` for a virtual thread's state is a type error under clang-cl:
pthread_t is a POINTER on Apple and glibc, but the Windows compat shim defines it
as struct {handle, id}, so the assignment reads as "assigning to 'pthread_t' from
incompatible type 'int'". memset over sizeof is correct for both shapes, and
gcPthreadValid -- set FALSE on the next line -- is what actually gates every read
of the field.
cn1AllocThreadStack declared its byte count above the #if that uses it, so on
Windows, whose arm calls calloc with the element count instead, it was an unused
local. Moved onto the arm that uses it.
Swept the rest of this branch's additions for the same class of thing rather than
waiting for CI to find them one at a time: every other POSIX call in code Windows
compiles is either guarded (mmap/munmap behind !_WIN32, pthread_attr_setstacksize
behind __linux__) or shimmed in cn1_win_compat.h (usleep, pthread_key_create,
pthread_getspecific). The virtual-thread runtime -- including the
__attribute__((weak)) definition, which clang-cl treats differently on COFF -- is
entirely inside the CN1_VIRTUAL_THREADS gate, which excludes _WIN32, so none of
it is compiled there at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A null array crashed instead of throwing (P1). CN1_ARRAY_STORE_CHECK evaluates CN1_CLASS_OF(arrayObj) with no null guard, and under -Dcn1.checkedCasts it runs AHEAD of the setter that turns a null array into a NullPointerException -- so an object-array store through a null array took the process down. Java orders NPE ahead of ArrayStoreException anyway, so falling through to the setter is both the safe answer and the correct one. A virtual thread's stack could go unmarked mid-switch (P1). The parked-stack pass skipped anything cn1VirtualThreadIsRunning() reported, on the reasoning that the carrier covers those. It does -- but only once the carrier's stack pointer is actually INSIDE the virtual stack, and `running` is raised before the switch and lowered after the switch back. In those two windows a stopped carrier still has an OS-stack pointer, so cn1VirtualThreadForStackAddress matches nothing, the carrier pass scans only the OS stack, and this pass skipped the virtual stack for being "running". References held in C temporaries there could be swept. The flag cannot be made atomic with the switch it brackets, because the switch is what changes the stack the flag would have to be written from. So the passes now OVERLAP instead of partitioning: every virtual thread's saved region is scanned unconditionally. Safe, because [sp, stackHigh) is inside the mapping whenever sp is non-zero; complete, because while a virtual thread runs the carrier's pointer is lower, so this pass covers a subset and the carrier covers the rest; and cheap, because conservative marking is idempotent. cn1RetireVirtualThread's "use after free" was NOT one, and the code now says so. markDeadThread -> collectThreadResources sets gcQueuedForDrain unconditionally and has no early return, so the synchronous release branch was unreachable. It read as live, though, so it is gone and the invariant is written down -- including the reason it matters, which the report had right: codenameOneGCMark copies each ThreadLocalData* out of allThreads under the critical section and dereferences it OUTSIDE the lock, so a synchronous free would be a genuine use-after-free. File.list returned something that called itself a String. All three arms passed the ELEMENT class to allocArray, which installs whatever it is given as the array object's own class; cn1MainArgs has always passed class_array1__java_lang_String. Pre-existing on iOS and Linux, copied into the new Windows arm, fixed on all three. Windows absolute paths were treated as relative, which corrupted them rather than merely misreporting them: getAbsolutePathImpl tested p[0] == '/', so "C:\data" had the working directory prepended. There is now a per-platform predicate that knows about drive letters and UNC roots. The matching Java-side gap is deliberately left and documented at the predicate: File.isAbsolute() tests startsWith(File.separator) and separator is "/" everywhere, which needs a per-platform separator in shared JavaAPI -- a change for every port, not for making the clean target build. Blocking file reads and writes now park the mutator, like the socket reads and StandardInputStream already did: a FIFO, a device or a network-backed path blocks for as long as the far end stays quiet, and an active thread there strands the collector waiting for a safepoint that cannot arrive. Both carry the buffer keep-alive for the same reason those do -- only an interior pointer is live across the call. (Moving that macro above its first use is why it now sits at the top of the file layer rather than beside stdin.) The benchmark helper compiles the emitted .S. Third place with this bug: the CMake generator and the Xcode project generator had it too, and a *.c-only invocation links against a missing cn1VirtualThreadSwitch on any target where the switch exists. Two findings are recorded in the file rather than fixed, with the analysis and the actual remedy: 32-bit ftell/fseek cannot express a position past 2GiB where C long is 32 bits, and paths reach the narrow CRT as UTF-8 and are read as ANSI. Both are pre-existing on every platform, both want a change across the whole file layer, and neither is what enabling the clean target is about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mapper.Direct promises identical output, not better output. Each of these was the
direct path being reasonable in a way emitFieldToMap is not, which is the same
thing as changing a mapper's wire format the day it gains a direct writer.
- A property NAME was escaped for the Java literal and not for JSON. escape()
doubles a quote so the generated source compiles; the resulting writer then
appended the raw character, so a @JsonProperty holding a quote emitted
"a"b" -- unparseable. The map path never had this because JSONWriter puts the
key through writeString. Now jsonEscape composed with escape: one makes the
JSON valid, the other makes the source compile. Done at generation time, since
a jsonName is a compile-time constant and the writer should stay a literal
append.
- A Property value was rendered too well. emitFieldToMap stores it RAW, so
JSONWriter renders a Date or a mapped object through String.valueOf;
appendJsonValue turned them into epoch millis and nested JSON. New
Mappers.appendJsonRaw is exactly JSONWriter's answer for a value that was put
in the map unchanged.
- A reference field looked its mapper up by RUNTIME class. A field declared as a
mapped base holding an unmapped subclass therefore found nothing and fell back
to a quoted toString, where the map path asks Mappers.get(Declared.class) and
serialises it as an object. New Mappers.appendJsonUsing takes the mapper the
caller names, and still uses that mapper's direct route when it has one.
- Mapped list ELEMENTS had the same problem, plus the general one behind it: the
direct path had a two-way branch where emitFieldToMap has four. It now mirrors
them one for one -- enum name(), scalar raw, Date getTime(), everything else
through the declared element type's mapper.
The test was the actual defect. Nothing compared the two paths against each other,
which is why all of this shipped; and the parity test added for the first pair
needed three fixes of its own before it proved anything:
- It went through Mappers.appendJson, which consults the registry. In an
isolated classloader the registry is empty, so it compared the map path
against "com.example.Swatch@23706db8". It now drives the generated writer.
- The polymorphic case had no mapper registered for the base type, so BOTH paths
fell back to toString and agreed. Registering it is what makes the two
implementations able to differ at all.
- assertEquals reports the FIRST difference, so one unfixed case masked the
others. Each representation is now pinned individually, which also catches the
case equality cannot: both paths wrong in the same way.
Verified by reverting the generator with the test in place: one failure against
the old code, six passing against the new.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two more review findings, both in code this branch touched. skip(Long.MAX_VALUE) computed `start + count` and clamped afterwards. Once any byte has been read that addition overflows signed long -- undefined behaviour, and in practice a wrap to negative, so the seek goes BACKWARDS and the caller is told it skipped a negative distance or gets an error where it should have landed on EOF. It now clamps against the remaining DISTANCE, which cannot overflow: end is at least start, and start plus the clamped amount is at most end. File.list walked the directory TWICE -- count, allocate, walk again -- and assumed both walks saw the same directory. They do not. A file created in between overruns the array, and CN1_SET_ARRAY_ELEMENT_OBJECT turns that into ArrayIndexOutOfBoundsException; a file removed leaves trailing nulls in a String[] that no caller expects. Directories change under readers routinely, so this was never sound. I wrote the Windows arm that way deliberately, mirroring the POSIX one, which means I copied the structure without asking whether it held. Both arms now enumerate ONCE into a small growable list of names and build the array afterwards. The names are held in C memory on purpose: allocArray and newStringFromCString can both collect, and nothing may hold a directory handle across that. The ObjC arm is left alone -- NSFileManager hands back a snapshot, so it never had the race. Also moves stdlib.h to the shared include group, since the list uses malloc/realloc/free on both arms and sits outside the platform blocks. The test is the part worth reading. FileClassIntegrationTest never called File.list(), so the native listing was COMPILED but never RUN by any suite: the rewrite above passed 5/5 while executing none of it, and reverting it would have passed too. Coverage now creates a directory, lists it, and pins the three things that were wrong or fragile -- the entries, the absence of nulls, and that the result is a String[] rather than a String, which is the pre-existing allocArray class bug nothing had ever asserted. Confirmed the assertions discriminate rather than merely execute: with the array class reverted to the element class, all five configurations FAIL; restored, all five pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two more findings, both consequences of this branch making java.io.File usable on Windows. "C:foo" is DRIVE-RELATIVE: relative to the working directory of drive C, which is not the process working directory and may be on a different drive. cn1FileIsAbsolute classified it correctly -- the comment there even says so -- and then the fallback prepended the process cwd anyway, producing "D:\cwd\C:foo", which names nothing. The predicate knew about a case the code after it did not. _getdcwd asks the right drive. Deliberately not _fullpath, which the report suggested: it also normalises "..", and getAbsolutePath is specified NOT to do that -- resolving is getCanonicalPath's job. Using it would have swapped a wrong path for a subtly wrong contract. createNewFile was check-then-act: access(), then fopen(p, "w"). Losing that race does not merely return the wrong answer, it TRUNCATES the file the other process just created, and then reports true as though it had done the creating -- which is exactly the failure mode the lock-file and single-instance patterns it exists for cannot survive. Now a single O_EXCL open on both arms, with the kernel deciding. Pre-existing on POSIX too, so both are fixed. ON THE TEST, because the distinction matters: the coverage added here is a REGRESSION GUARD, not a demonstration of atomicity. It checks the uncontended path -- createNewFile on an existing file returns false and leaves it intact -- and the old check-then-act version passes it too, because access() succeeds and it returns before reaching the truncating fopen. Confirmed by running the suite against the old implementation: 5/5 green. The real defect needs a file to appear between the check and the open, which one thread cannot arrange, so the argument for the fix is structural rather than empirical and the comment in the test says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
newStringFromCString turns each byte into its own char. That is correct for what it exists to serve -- generated string literals, which are ASCII plus ~~uXXXX escapes -- and wrong for anything arriving from outside the program. A UTF-8 "e-acute" is two bytes, so main(String[]) and System.getenv handed back one garbage char per byte, corrupting paths and option values before the program had a chance to look at them. Both entry points are new in this branch. newStringFromUtf8 decodes properly: multi-byte sequences, surrogate pairs for astral code points, and U+FFFD for malformed input the way java.lang.String's own decoder does -- a program should not die because one environment variable holds a stray byte. Overlong forms, UTF-8-encoded surrogates and out-of-range code points are all rejected. newStringFromCString itself is deliberately NOT changed. Every native-to-Java string in the VM goes through it, its byte-widening is load-bearing for the literals it serves, and its own comment records that the high-bit path is bit-identical to what came before. Correcting the two entry points this branch added is the scoped fix; the general version is the same work as the ANSI-versus- UTF-8 path issue already recorded in nativeMethods.m. TWO BUGS UNDERNEATH, both found by the test rather than by reading: newString was broken and had never been called from C. JAVA_CHAR is an int and JAVA_ARRAY_CHAR is an unsigned short, and it sized the allocation with sizeof(JAVA_CHAR) while memcpy'ing length * sizeof(JAVA_ARRAY_CHAR) bytes out of a four-byte-element array -- half the input, at the wrong stride. My decoder was its first caller and hit it immediately: "cafe" came back as c,NUL,a,NUL,f. It now narrows element by element. Behind that, the representation is not a free choice. A string whose units all fit in a byte is stored as a COMPACT byte[], anything else as a char[], and charAt reads whichever it finds -- so handing it the wrong one reads 8-bit units out of 16-bit data and produces exactly the same symptom rather than failing. That rule now lives in cn1StringFromUnits, used by newString and newStringFromUtf8. newStringFromCString keeps its own copy on purpose: it tracks the Latin-1 flag during decoding and runs for every literal at startup, so routing it through a helper that recomputes would add a pass over every literal in the program to save a dozen lines. The comment says so, and says the two must change together. The test reports CODE POINTS rather than text, so it cannot pass through a console-encoding coincidence: "cafe-acute-euro" must arrive as 99,97,102,233,8364, which covers a two-byte and a three-byte sequence. Byte-widening reports the individual bytes instead, which is how the newString bug surfaced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s UTF-8 The Windows clean-target leg failed the test added with the UTF-8 decoder, and it was right to: "cafe-acute-euro" arrived as 99,97,102,65533,65533 -- c, a, f, and two replacement characters. The CRT hands main() and getenv() the wide command line and environment already converted down to the ACTIVE CODE PAGE, so decoding those bytes as UTF-8 finds invalid sequences and substitutes U+FFFD for every non-ASCII character. That failure was predicted by a comment I had written in this very function -- which then shipped alongside a test asserting the behaviour the comment said did not exist. MultiByteToWideChar with CP_ACP is the conversion Windows actually needs, and it yields UTF-16 code units directly, so nothing decodes afterwards. RENAMED from newStringFromUtf8 to newStringFromNative for the same reason: a function named FromUtf8 that deliberately does not decode UTF-8 on one of its platforms is a trap for whoever reads it next. The name now says what it does -- convert text that came from the OS, in whatever encoding the OS used. WIN32_LEAN_AND_MEAN before windows.h, which is the same winsock timeval collision that broke java_io_File.m; and the byte-length local moved onto the POSIX arm, which is the only one that uses it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The JLS orders these: NullPointerException, then ArrayIndexOutOfBoundsException, then ArrayStoreException. Under -Dcn1.checkedCasts the emitted covariance check ran BEFORE the setter that reports the first two, so a store with both a bad index and an incompatible value reported the value -- hiding the exception the program should have seen. (The null case was worse and is already fixed: the check dereferenced the array to reach its class.) The store check is now guarded by the same access validation the setter performs, so the first two exceptions are thrown first and in the right order. The setter re-checks, which on the in-bounds fast path costs one comparison. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e collector This backs out my own fix from earlier in this branch. Marking the attached ThreadLocalData threadActive around the context switch reads as obviously correct and is a REGRESSION, worse than what it fixed. A virtual thread's state has no pthread of its own -- deliberately, it may run on a different carrier next time. The collector's wait for a lightweight thread is `while(t->threadActive) usleep(500)` with no bound, and the forced-stop escalation that exists to break exactly that wait is gated on gcPthreadValid, which is permanently false here. So the flag converts a POSSIBLE race on the state's object stack into a CERTAIN hang for any virtual thread that computes without reaching a safepoint: the collector waits for a flag only that thread can clear, and cannot stop it. What the same report asked for has two halves, and the other one stands. The C stack is covered: cn1GcScanParkedVirtualThreads scans every registered virtual thread whether or not it is running, so no virtual stack goes unscanned during the windows where `running` is set but the carrier has not switched yet. That fix is independent of this revert and stays. The half that remains open -- a collection walking the state's object stack and pending-allocation table while the virtual thread mutates them -- is documented at cn1SpawnVirtualThread along with why the obvious fix is worse and what the real one is: carrier association. A running virtual thread executes ON a carrier that does have a stoppable pthread, so the collector should satisfy the wait by stopping the carrier. That needs the stop handshake to stop being per-TLD (the signal handler records into the TLD of the thread it runs on, which is the carrier's), i.e. a change to the collector's stop protocol rather than to the spawn path -- not something to improvise in an API that has no callers yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With -Dcn1.checkedCasts the covariance check broke correct programs, which is the worst direction for a check to fail in. A generated array class records arrayType as the BASE element class rather than the immediate component: String[][] has dimensions 2 and arrayType String, not String[]. So `values[0] = new String[1]` asked whether a String[] is an instance of String, got no, and threw ArrayStoreException on a store the language requires to succeed. Restricted to dimensions == 1, where arrayType genuinely IS the component type. Multidimensional stores lose a diagnostic that did not exist before this feature was added; the alternative was breaking working code. Covering them properly needs the immediate component type, either emitted per array class or reconstructed from dimensions at runtime, and the macro says so. Also fixes a timeout in VirtualThreadRuntimeTest that could never fire. It read the child's output inline and then called waitFor: the read blocks until the child closes stdout, so a binary that hangs -- exactly what a context-switch regression produces -- never reached the timeout, and the Maven job would sit until CI killed it instead of the test failing. Output now drains on its own thread, with a bounded join so a wedged reader cannot reintroduce the hang the change removes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This corrects my own change earlier in this branch, and the reasoning behind it was the defect. "A thread it cannot stop is one it does not scan either way" is true only while the thread genuinely cannot be stopped. Failures are often TRANSIENT -- a stop signal briefly masked is enough -- and the thread recovers. Skipping it then meant cn1GcScanThreadNativeStack returned without scanning a RESPONSIVE thread, for roughly the next sixty collections, so references held only in frameless C locals or registers went unmarked and could be reclaimed while still in use. A GC correctness bug, traded for a performance win. The two things I had conflated: the cost was never the SIGNAL, it was the WAIT. One unresponsive thread consumed the entire 2,000,000-spin budget -- 267ms of a 280ms mark. So a thread with a failure history is now probed with a 20,000-spin budget rather than skipped. Healthy threads answer within about 200 spins, which is a hundredfold margin for one that is merely slow, at one percent of what a hang used to cost; and a thread that recovers is picked up on the very next cycle instead of up to 64 later. Verified across the GC suites, including GcUncooperativeThreadIntegrationTest -- the issue #5537 scenario this logic exists to serve: 6/6. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Boolean shares its kind with boolean and Character with char, so the direct writer treated both as primitives. Only the boxed form can be null, and both handled it wrongly in opposite ways: a null Boolean was unboxed by a ternary and threw NullPointerException, and a null Character went through String.valueOf(Object), which returns the four characters "null", and was then QUOTED -- so an unset field serialised as the string "null". The map path stores the value and lets JSONWriter see the null, emitting JSON null for both. Told apart by binaryName, which does distinguish them, with a temporary in each so a getter is not evaluated twice, and charValue() so String.valueOf resolves to the char overload rather than the Object one. The parity test carries both fields now, and they discriminate by construction: against the old code the Boolean case throws (a test error) and the Character case produces a quoted "null" against the map path's null (an assertion mismatch). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CODENAME_ONE_ASSERT is plain assert(), which NDEBUG compiles out of every release build. So once all NUMBER_OF_SUPPORTED_THREADS slots were taken, threadOffset stayed -1, the assertion vanished, and the next statement executed allThreads[-1] = i -- writing over whatever precedes the table. A debug build aborted; a shipped one carried on with silent memory corruption, which is the worse of the two. Capacity exhaustion is a condition to report, not to assert. It returns 0 now, and cn1SpawnVirtualThread already checks for that. Pre-existing rather than new: every OS thread creation runs this path too. A virtual thread per request only makes reaching the limit realistic. The partially built state is unwound through cn1FreeThreadLocalDataFields, extracted from cn1ReleaseThreadLocalData rather than copied, because the release path also decrements nThreadsToKill and a state that never reached allThreads was never counted as living. Duplicating the frees would have drifted apart, and getting that counter wrong would have been a slow leak in the opposite direction. Verified across the GC suites including GcUncooperativeThread and GcHeapIntegrity: 6/6. (The translator build says nothing about this -- it compiles Java, and the C here is only compiled by those tests.) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…reeing Two defects, and both are mine from earlier in this branch. THE HANG I REVERTED WAS STILL REACHABLE. Removing the threadActive assignment from cn1VirtualThreadResume did not close it, because CN1_RESUME_THREAD does the same thing and every bracketed native goes through that macro. getThreadLocalData() resolves to the VIRTUAL thread's state while one is running, so a virtual thread that read a file or a socket returned with its state marked active, and nothing lowers it again until the next yield. Same unbounded while(threadActive) wait, same forced-stop escalation gated on gcPthreadValid and therefore unavailable, same stall. I checked the call site I had edited and not the shared path through it. The guard states the invariant the code always needed: mark active only what the collector can STOP. gcPthreadValid is exactly that question. A real thread is unaffected; a virtual thread's state stays down, which is where it was before any of this. Roots do not depend on the flag -- cn1GcScanParkedVirtualThreads scans every registered virtual thread whether or not it is running. THE EXHAUSTION CHECK INTRODUCED A USE-AFTER-FREE. pthread_setspecific binds the new state to TLS above the capacity search, so the failure path I added freed a state the key still pointed at: every later getThreadLocalData() on that thread would return memory that had been given back. That is worse than the out-of-bounds write it replaced, because the thread keeps using the stale pointer rather than failing. Unbound before the free. Also: System.getenv(null) throws NullPointerException as the API requires, instead of returning null and making an invalid argument indistinguishable from an unset variable. Verified across the GC suites, 6/6, including GcUncooperativeThread and GcHeapIntegrity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0b22afd0fb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The HTTP/2 descriptor cap was consulted AFTER the submission it was meant to prevent. The turn check stops the session that crossed it, but every other session wakes on a control frame and submits one more first, so the real bound was the cap plus one per connection -- and a peer holding its window shut can keep waking them. It is consulted before submitting now. Answered rather than deferred, because the handler has already opened the descriptor: holding the response would hold the very thing being rationed, so it is closed and the stream gets a 503. The Lambda loop had a third way to leave an invocation unresolved. Two branches now stop when the failure cannot be reported; the branch where the API REFUSES a result -- a 413 for an oversized payload is the ordinary case -- ignored the same boolean and polled on. And a contract endpoint's short, byte or float parameter was parsed wide and cast down, which wraps rather than fails: "40000" for a short reached the handler as -25536, "256" for a byte as 0, and 1e100 for a float as infinity. These are client-controlled values. They are parsed at their own width now, so an out-of-range one is a 400 like any other unparseable number -- which is what the BOXED forms beside them always did, since Short.valueOf throws. Only the primitives were cast. Verified: 42 processor tests, 33 HTTP tests and 2 Lambda integration tests pass, with the native verifier strict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99a14f4df2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…refix A Float was widened to double by the ByteSink writer and printed with the double's spelling, so 1.2f went out as 1.2000000476837158; the String writer calls Float.toString and sends 1.2. HTTP/1.1 writes through the first and HTTP/2 through the second, so ONE handler answered two different numbers depending on which protocol the client negotiated. It keeps the value's own spelling now. This is the second defect of exactly that shape -- byte[] was the first, and its branch already carries the warning -- so the selftest now COMPARES the writers over floats, doubles, longs past 2^53, ints, booleans, a String and a byte[], rather than asserting either one's output. That is the check that would have caught both. With the fix reverted it reports "expected <1.2> but was <1.2000000476837158>" and three more. Separately, a relative class-level @RequestMapping("api") produced the route "api/users". Every request target starts with "/", so nothing could ever match it and the endpoint answered 404 from a build that reported success. The method-level path was normalised this way already; the class-level one was not. Verified: 27 controller processor tests, 33 HTTP tests, and the selftest on both runtimes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b45ff6d94
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The @RestController half was taught two weeks of lessons that this half never heard. A literal beside a placeholder was refused. The dispatcher emits every route without a placeholder before every route with one, so /users/me beside /users/{id} is decided by that order -- the literal takes its own path and everything else falls through. The comment on the check even says literal-first cannot break a tie between two DYNAMIC shapes, which is true, and equally means it DOES break this one. Two dynamic shapes still clash. A placeholder nothing binds was accepted, and that is worse than a typo: the client substitutes the placeholder's own NAME, so it asks for /users/id literally, while the server matches any value there and passes it to nobody. Both halves compile and agree on a route whose variable can neither be supplied nor read. The @Path-to-placeholder direction was checked; this is the reverse. StaticFiles refused a verb before deciding the path was even its own, so a POST to an unrelated path came back 405 instead of reaching the 404 the caller meant -- and in a chain that tries files first it would shadow a later dynamic handler entirely. The mount is checked first now. Verified: 18 contract processor tests pass, and reverting either fix fails its own test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # scripts/cast-semantics-baseline.txt
… rescoped Master scoped the cast-semantics gate to what ParparVM actually translates, so this port is no longer scanned and the comment's reason for the instanceof was out of date. The guard itself stays: the extra is whatever the sending application put there, so the cast really can fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ca4b7281d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
AndroidImplementation.java is a CRLF file. Editing it with a script that reads and rewrites the whole text normalised every line to LF, so a twelve-line change was recorded as 18,364 added and 18,354 removed. It is rebuilt from the committed bytes here with the same twelve lines applied, and the diff is 12/2 again. This is not cosmetic. CodeQL's analysis is diff-informed, so a file that appears wholly rewritten is treated as wholly new: 19 path-injection alerts were raised against this PR, ten of them in this file, for code it never touched -- and the same rule in the same file is already dismissed on master. The finding was mine to cause and mine to undo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
check-native-signatures could report success having checked the backend against NOTHING. When the ParparVM sources fail to compile the classes path is left empty -- deliberately, so the entry would skip -- but an empty path expands to "$REPO_ROOT/", which IS a directory and passes every test: the entry took the repository root as its classes, found no natives to disagree with, and printed the same "== backend" as a real pass, while --require-all counted it as covered. The comment above it already claimed the behaviour that was never implemented. An empty path is now a port that could not be built. Demonstrated both ways, and the broken half was not hypothetical: this machine's tools/env.sh JDK is missing currency.data, so the backend compile really does fail here. With it, the gate now skips loudly and --require-all exits 2; with a working JDK all five ports including the backend are checked and it exits 0. A contract body typed Map<String,Integer> was accepted. A map's VALUES are handed over as the parser built them -- nothing walks them applying the declared type, the way collection elements are walked -- and the parser answers Long for every JSON integer, so the map is a map of Long and the handler's first read as an Integer throws. Only the types the parser really produces may be declared. Note the first rule I wrote for this was too strict, and five existing tests said so: this half GENERATES codecs, so List<Dto> and typed collections do decode. The rule is now the narrow one the defect actually describes. And a DTO's float field took a plain cast, which saturates: a finite 1e100 became infinity, a value JSON cannot express and the client did not send. Range-checked now, like the scalar text path -- otherwise the same value is accepted or refused depending on where it appears. Verified: 50 processor tests, and reverting the map rule fails its test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 44603ebc71
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… unreadable Float.parseFloat does not FAIL on a value too large for a float -- it answers infinity. So a controller's float path, query or header parameter took 1e100 straight past the generated guard and the handler ran on a number the client never sent. Every other width throws, which is why only this one slipped: the guard was right for five types out of six. With the check reverted the new test reports "expected:<400> but was:<200>". An input that really spells an infinity is still accepted, which is what parseFloat means by it. That is the third place this exact narrowing has bitten -- the contract's scalar text path, a DTO's float field, and now the controller's guard. Separately, and NOT fixed here: staging resources for translation does not make them readable. The backend translates as app type "clean", and only the linux and windows types embed classpath resources into the binary, while the clean runtime's Class.getResourceAsStream returns null unconditionally. So getResourceAsStream finds the file under cn1:backend, on the JVM, and finds nothing in the packaged executable -- which is the same silent divergence the staging step was added to close, one layer down. Packaging now says so, naming the count, instead of leaving it to be found in production. Embedding them properly is a translator change plus a native plus a hook in Class, which vm/JavaAPI shares with iOS. That belongs in its own change with its own review, not at the end of this one. Verified: 51 processor tests, and reverting the float guard fails its test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0c1845887f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… one A failed malloc for the request body sent the request ANYWAY. The POSTFIELDS block is skipped when the copy is null, so an allocation failure produced the same request with an empty body and reported success -- S3's putObject would replace the object with nothing and tell the caller it worked. A request whose body could not be made is now a failed request. CURLOPT_TIMEOUT caps the WHOLE transfer, so an upload or download that was progressing perfectly well was aborted at thirty seconds for no reason but its size. The Java SE arm sets a READ timeout, which fires only when a single read stalls, so an object large enough to take half a minute transferred under cn1:backend and failed once packaged. Replaced with a connect deadline and a stall deadline, which is libcurl's spelling of the same rule. And a multi-range request was answered 416. That status asserts that NONE of the requested ranges exist, and "bytes=0-99,200-299" over a 256KB file is entirely satisfiable -- this server just does not assemble multipart/byteranges. Not being able to honour a Range is not the same as the Range being unsatisfiable: RFC 9110 14.2 says to ignore the field and send the whole representation, which every client understands. An unparseable Range is ignored for the same reason. A range that really cannot be satisfied is still 416, and its test still passes. Verified: 33 HTTP tests with the native verifier strict; reverting the range rule fails with "expected: <200> but was: <416>". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 19ec6baed2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The HTTP/2 body budget was per SESSION while the descriptors beside it were counted per process, and the descriptors were right. A session pausing itself after one oversized body still lets the process hold that body times every connection, and the connection ceiling is in the thousands: small GET requests from peers that never open their windows could pin gigabytes of native memory. Counted across the process now, at the three points that already existed -- submitted, drained, freed -- so the figure cannot drift from what the bodies really hold. Http1Date accepted dates that are not dates. Every field is read at a fixed offset and handed to a civil-date routine that NORMALISES whatever it gets, and nothing checked the suffix or the ranges, so "Sun, 99 Nov 9999 99:99:99 BAD" parsed to the year 9999 -- and StaticFiles read that as newer than the file and answered 304, sending no content to a client that had nothing cached. With the check reverted the selftest reports "expected <-1> but was <253405860039000>" and three more like it. The whole IMF-fixdate shape is verified now, including that a day exists in its month; the one real form still parses and still formats back. And a contract Map<Integer,String> was accepted. A JSON object's names are strings, always, so integer keys arrive as String: a lookup finds nothing and iterating the entries throws, while encoding turns them back into strings. The value half was already checked; the key half was not. Verified: 34 backend tests with the native verifier strict, 48 processor tests, and the new native symbol is checked -- giving it the wrong return token fails the build naming it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d8abc376ee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The review found one toLowerCase() on a protocol token, in the SigV4
canonical headers. There were SEVEN in this branch's code, and the repo has
a standing rule about exactly this, so they are all fixed together rather
than one per round:
Aws signed header names -- a Turkish locale folds If-Match to a
dotless i, so the canonical request stops matching what AWS
computed and every such request is rejected as a bad signature
Jwt the "bearer " scheme prefix -- folded, it stops equalling the
constant, and EVERY bearer token is refused
StaticFiles the file extension that keys the MIME table -- ".PNG" would
not find image/png
Web x2 arms header names, both storing and looking up -- getHeader answers
null for a header that is present, in both arms
Jwt takes regionMatches(true, ...) rather than a fold: it compares character
by character, is locale independent, and allocates nothing. The rest get the
six-line ASCII fold the tree already carries in four other classes -- copied
rather than shared, as CLAUDE.md says.
Two decode fixes as well. A @Body Collection<Note> fell through to a guarded
cast because only List and Set were recognised as collection shapes, so the
handler got a collection of Map and threw on its first element; the three
are now one predicate, since any place that lists two of them and not the
third has the same hole. And a numeric annotation default that is not a
number bound ZERO -- defaultValue="oops" on an int -- while the non-empty
default also suppressed the required-value guard, so an absent parameter
reached the handler as a value nobody wrote. That is the author's own
configuration and is now refused at build time.
Verified: 51 processor tests, 34 backend tests, and the selftest checks the
bearer fold on both runtimes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8141d2d70d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The INBOUND ceilings were per session, the same asymmetry the response bodies had a round ago. A session may hold 32MB of request buffers within its own limit, and the connection ceiling is in the thousands, so a handful of clients keeping streams just under it exhaust the machine while every session stays honest. Headers and bodies are both counted across the process now, released in cn1H2FreeRequest, which is the one place a request's memory goes away. The two charges are shaped differently on purpose. A header field is charged unconditionally, because r->headerBytes above it already counts those bytes and the free gives back exactly that -- the two have to move together. A body chunk is TESTED first and charged only after the append succeeds, because the free gives back bodyLength: charging first would strand the bytes of an append that then fails to grow the buffer, and the counter would drift up until it refused everything. The load-then-add can overshoot by a chunk when two sessions cross together, which is the right trade for a coarse memory guard against a lock on the data path. And -Dcn1.backend.sqlite=false could not produce a binary. Turning the engine off is two changes: without -Dcn1.sqlite=true the translator leaves cn1_sqlite3.h out, but cn1_backend_db.c is compiled either way and its SQLite half includes that header at line 133 unless CN1_BACKEND_NO_SQLITE compiles it to stubs instead. build.sh has always set both; the Maven goal set only the first, so the option advertised as saving the engine failed at the C compile. Verified from the other direction: the script's path, which sets the macro, builds the selftest with the engine off and produces a working 3.1MB binary. The third finding in this round -- locale folding in Web -- was already fixed in 8141d2d; the review ran against an older commit. Verified: 34 backend tests with the native verifier strict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1cdb850496
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
An HTTP/1 request body was allocated at its DECLARED length before a byte of it had arrived. Content-Length is a claim, and this loop then holds that memory until the rate allowance expires -- so an unauthenticated client sends a header and nothing else, and the server reserves 8MB on its word. The connection ceiling is in the thousands. The buffer grows toward the declared length as the bytes ARRIVE instead, which is the only figure a client cannot lie about. Doubling is what keeps that affordable: growing by each read's size was the original defect here, about a thousand resizes and 4GB of copying for one 8MB upload. Every growth is capped at the declared length, so the last one lands exactly on it and the invariant the rest of the class depends on -- buffer.length means "bytes readable" -- is unchanged. A body smaller than the starting chunk still takes a single exact allocation, as before. No global counter for this, deliberately. A reservation would have to be threaded through borrowed thread buffers, owned copies and every failure path, and one leaked reservation wedges the server for good -- a worse failure than the one being fixed. Growing with the data needs no counter at all. The HTTP/2 side had the ordering wrong for the same reason: the process-wide check sat AFTER h2.respond(), which is the call that copies the body into native memory, so it had already spent what it was meant to withhold -- and every session wakes on a control frame and spends one more. Checked before the copy now, answering 503 rather than copying. And a map value declared as a nested container was accepted: Map<String, Set<String>> passed because only the outer raw type was tested, while a JSON array always arrives as a List and nothing converts a map's values at any depth. Every level is now checked against what the parser really hands over -- which is Map and List, not Set or Collection. Verified: 33 HTTP tests with the verifier strict, including the 8MB upload and the partial-request fixtures, and 51 processor tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e954b461a7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
"?limit=" is a parameter the client SENT, and queryParam distinguishes it from one that was left out -- but the generated guard treated null and empty alike, so an empty value passed as valid and to<Type> substituted the default or zero. The handler then ran on a number nobody wrote, which is the same defect as accepting "zz" for an int, and that has answered 400 since the guard was added. Only ABSENT bypasses parsing now; the declared default still applies when the parameter really is omitted, which the test checks in both directions. With the change reverted it reports "expected:<400> but was:<200>". Also removes a stray javadoc left above the wrong method by an earlier edit in this branch. Verified: 55 processor tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ecfc128a0a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The process-wide counter could be driven DOWNWARD by rejected headers. r->headerBytes is incremented at the top of the callback, but the global was charged after the per-stream and per-session checks -- so a header block that breached either was counted by headerBytes, never added to the global, and then SUBTRACTED from it by cn1H2FreeRequest when the stream was reset. Repeat that and the total goes negative, at which point the cap it exists to enforce admits everything: a client can spend oversized rejected header blocks to buy room for request bodies it would otherwise not be allowed to hold. Charged in the same breath as r->headerBytes now, ahead of every return, so the two figures move together whichever way the stream ends. That is the rule the body path already follows from the other direction -- tested first and charged only after bodyLength grows, because there the free gives back bodyLength. Audited the invariant rather than just this path: the global is the sum over live requests of bodyLength + headerBytes. Neither field is ever reset (both only ever grow), every charge sits with the growth it accounts for, and cn1H2FreeRequest is the only thing that destroys a request. Verified: 33 HTTP tests with the native verifier strict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 95ea0de0d7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…or chose The inbound budget counted what ARRIVES in a request and not the request itself. CN1H2Request embeds CN1_H2_MAX_HEADERS slots, so one is about a kilobyte before a single header byte is read -- and a client that opens the advertised stream concurrency with minimal headers keeps every payload counter near zero while holding one per stream, per connection. It is a fixed cost per open request, so it is charged as one, at the allocation and released in cn1H2FreeRequest with the rest of what the request holds. The stream is refused rather than the connection, which is the proportionate answer to a process that is momentarily full. Separately, cn1:backend scanned for main methods while ignoring the entry point the generator had already chosen. Annotation processing writes it to META-INF/cn1-backend-main and cn1:backend-package reads it; the run goal did not, so a module holding any demo or tool with a main was refused as ambiguous even though the choice was made and recorded. The marker is consulted first now, and the scan stays for modules written by hand, which have no marker. Verified: 33 HTTP tests with the native verifier strict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a522d42094
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| #ifdef CURLOPT_PATH_AS_IS | ||
| curl_easy_setopt(curl, CURLOPT_PATH_AS_IS, 1L); | ||
| #endif |
There was a problem hiding this comment.
Enable CURLOPT_PATH_AS_IS with a real version check
When the packaged runtime sends an S3 request for a key containing a dot segment, such as a/../b, this guard is always false because CURLOPT_PATH_AS_IS is an enum member declared through CURLOPT(...) in curl.h, not a preprocessor macro. Libcurl therefore normalizes the URL to /b while Aws signs /a/../b, causing SignatureDoesNotMatch and diverging from the Java SE runtime; use a LIBCURL_VERSION_NUM check or enable the option unconditionally for the bundled supported versions.
Useful? React with 👍 / 👎.
Adds a server-side runtime that runs a Codename One handler through the ParparVM
pipeline: Java or Kotlin translated to C and compiled into one static native
executable with no JVM under it. About 8 MB, a few milliseconds to first
connection, about 3 MB idle.
What this is for, and what it is not
It does not replace Spring Boot, Jakarta EE, Quarkus or Micronaut, and it is not
trying to. Those carry a container, an ORM, a security stack and twenty years of
operations; none of that is here or planned.
It targets the region where the JVM's assumptions stop paying: cold starts
charged per invocation, baseline memory charged for an instance's life, sidecars,
edge locations, short-lived processes. That is where Java is thin and Go and
Node dominate, and where a Java shop ends up carrying a second language and a
second copy of every model that crosses the boundary. Either as a piece of a
larger deployment or as the whole server for a small project.
The vertical integration is the other half: one
@RestClientinterface generatesthe app's asynchronous client and the backend's synchronous half plus its
dispatcher, so a contract change is a compile error rather than a response the
app fails to parse in the field.
Where it stands against Go
vm/backend/benchmarksholds the harness. Two pinned cores, 64 connections,interleaved with rotating arm order, against fasthttp:
The /json figures are the generated-DTO path answering off a pooled response.
A handler that returns a
LinkedHashMapper request is about 0.58x, which thebenchmark keeps as its default because that is the honest cost of that shape.
Notable changes outside vm/backend
cn1_globals.mgainscn1SatbTrim. The SATB write-barrier log and its stagingbuffer only ever doubled and were never given back, so a process that saw one
busy period kept the peak for life -- 8 MB of a 12 MB plaintext process was an
empty buffer. Trimmed in the sweep against the recent high-water mark. This
reaches every Codename One target, not just the backend.
maven/pom.xmlbuildsmaven/backend, which was in no<modules>block, sonothing built the artifact
BackendPackageMojoresolves at run time.cn1:backendandcn1:backend-package, and the@RestClientserver-half processor.
backendmodule, behind-Dcodename1.platform=backendso a client-only app pays nothing for it.Testing
BackendHttpIntegrationTest21/21, plus the database and JavaSE-runtime suites.GcHeapIntegrity,GcOverflowSpiral,GcUncooperativeThread,LargeArrayGc,BibopPageFloor.GcSteadyState's 768 MB ceiling scenario fails on the dev machine and failsidentically with the SATB change stashed (895.8s against 913.7s, same timeout,
same scenario), so it is the known local failure rather than a regression. It
is
@Tag("benchmark")and runs in the benchmark job.--failure-level WARN,structure, cross-references, snippets, links, paragraph capitalization.
codenameone-maven-plugin: 0 findings. Copyright, controlcharacters and cast-semantics gates clean over the branch.
backend module compiled against
codenameone-backend.PMD and Checkstyle were not run locally; CI is the first run for those.