Make the frontend reusable in one process by xoxorwr · Pull Request #23868 · dlang/dmd

xoxorwr · 2026-09-16T19:41:53Z

This PR enables the frontend to be run in the same process without leaking memory & state between runs.

No behavior change for normal one shot compilation.

Each commit is independently reviewable.

Disclaimer: This is done with the help of a LLM.

…runs

rikkimax · 2026-09-16T19:48:20Z

 }

+// Initialize the identifier pool
+shared static this() nothrow


????

Modules constructors for this type of work, shouldn't exist, outside of a single central one.

Otherwise things will get missed over time.

True, moved it into Id.initialize()/Id.deinitialize(), and dropped module constructor.

Wait, that ctor is pre existing, I was only extracting them so they can be repopulated between runs.

If you can think of a better place for them, please let me know, I can't find where a central way would fit, there is some issues right now with how dmd expects things to init due to how unittests are called.

dmd/compiler/src/dmd/tokens.d

Lines 595 to 605 in 278dce2

// Initialize the identifier pool

shared static this() nothrow

{

Identifier.initTable();

foreach (kw; keywords)

{

//printf("keyword[%d] = '%s'\n",kw, Token.tochars[kw].ptr);

Identifier.idPool(Token.tochars[kw], kw);

}

}

thewilsonator · 2026-09-17T05:55:54Z

     */
    static IntegerExp createBool(bool b)
    {
-        __gshared IntegerExp trueExp, falseExp;


you can in theory do this for true/false as done here

thewilsonator · 2026-09-17T05:56:16Z

     */
    static IntegerExp literal(int v)()
    {
-        __gshared IntegerExp theConstant;


but not this, because it is inside a template.

though git grep \.literal\! shows only hits for IntegerExp.literal!0 and IntegerExp.literal!1

thewilsonator

I would try to split this up into separate PRs, much easier to review.

thewilsonator · 2026-09-17T05:58:15Z

 /***********************************************************
 */
+// Auto-incremented package tag; reset between sessions.
+private __gshared uint packageTag;


make this a static private member of Package

rainers

Although Visual D's semantic server tries to work incrementally to some extend (keep syntax-tree of parsing, don't ever release identifiers from the identifier pool, keep types that never change), keeping memory consumption low and free from false pointers is also a considerable advantage. Not sure how well this works for some parts of the semantic server that can currently still run in parallel.

It would be nice if you could split this into a couple of PRs

make local __gshared variables resettable
remove phobos from frontend.d (not used by Visual D). I suspect some of the reimplemented functions are also available in root/filename.d.
add the scratch allocator: not sure if we should add another allocator. Maybe it is easy to add free/realloc support for explicit calls in the exisiting bump-allocator. Note that D array allocation is used and is performed by the (usually disabled) conservative GC to allow requesting array metadata.

rainers · 2026-09-17T12:31:04Z


    static void* xmalloc_noscan(size_t size) pure nothrow
    {
+        if (isArenaEnabled)


Instead of testing isArenaEnabled it seems you can also just enable the GC and initialize arenaBit to zero or non-zero. That way the normal path does not receive an additional if.

rainers · 2026-09-17T12:39:00Z

+            if (pred(sv))
+            {
+                // Clear the value so the GC-scanned pool doesn't keep it reachable.
+                sv.value = T.init;


I don't think this works good enough, because the chain of used entries with hash collisions is interrupted. You need a separate marker for "deleted", see https://github.com/rainers/dmd/blob/dmdserver/compiler/src/dmd/root/stringtable.d#L217

rainers · 2026-09-17T12:43:46Z

+import dmd.dtemplate : TemplateDeclaration;
+
+// Cache for object._arrayOp; hoisted so deinitialize can reset it.
+private __gshared TemplateDeclaration arrayOpTemplate;


Your extracted variables seem very much in line with what Visual D does, though it abuses the mangling to access them, see https://github.com/rainers/visuald/blob/master/vdc/dmdserver/dmdinit.d#L68
Getting rid of these would be nice.

There are a couple of recursion checks using static nested or similar, though, which might also not get reset on fatal errors.

rainers · 2026-09-17T12:45:27Z

 }
+
+/// Reset the module's global state between analyses.
+void deinitialize() nothrow


Although precedent for deinitialize exists, I'd rather call these methods reset or resetState. Otherwise I would expect to have to call some initialize function on reuse.
Maybe too disruptive for existing methods, though, and inconsistent if only used for new ones.

xoxorwr added 4 commits September 16, 2026 20:09

Remove dependency on phobos

f579f4c

Add the mechanism to reset frontend state between analyses

9f5ded3

Add scratch allocator so non gc tracked leaks can be cleared between …

ff72172

…runs

Add arena support and reset remaining session state

e825d13

xoxorwr marked this pull request as draft September 16, 2026 19:43

rikkimax reviewed Sep 16, 2026

View reviewed changes

rikkimax requested a review from rainers September 16, 2026 19:48

xoxorwr added 2 commits September 16, 2026 22:22

Add Mem.enableZero to zero scanned allocations

cf83c43

Move identifier pool init out of a module constructor

a853da9

thewilsonator reviewed Sep 17, 2026

View reviewed changes

rainers reviewed Sep 17, 2026

View reviewed changes

0xEAB added AI Generated Code that is generated by an LLM AI. Severity:Enhancement DMD as a library instead of a standalone executable labels Sep 22, 2026

Navigation Menu

Uh oh!

Make the frontend reusable in one process - #23868

Make the frontend reusable in one process#23868
xoxorwr wants to merge 6 commits into
dlang:masterfrom
xoxorwr:lsp-fixes

xoxorwr commented Sep 16, 2026

Uh oh!

rikkimax Sep 16, 2026

Uh oh!

xoxorwr Sep 17, 2026

Uh oh!

xoxorwr Sep 17, 2026 •

edited

Loading

Uh oh!

thewilsonator Sep 17, 2026

Uh oh!

thewilsonator Sep 17, 2026

Uh oh!

thewilsonator Sep 17, 2026

Uh oh!

thewilsonator left a comment

Uh oh!

thewilsonator Sep 17, 2026

Uh oh!

rainers left a comment

Uh oh!

rainers Sep 17, 2026

Uh oh!

rainers Sep 17, 2026

Uh oh!

rainers Sep 17, 2026

Uh oh!

rainers Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	// Initialize the identifier pool
	shared static this() nothrow
	{
	Identifier.initTable();
	foreach (kw; keywords)
	{
	//printf("keyword[%d] = '%s'\n",kw, Token.tochars[kw].ptr);
	Identifier.idPool(Token.tochars[kw], kw);
	}
	}

Uh oh!

Conversation

xoxorwr commented Sep 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xoxorwr Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thewilsonator left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rainers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

xoxorwr Sep 17, 2026 •

edited

Loading