Proposal: minimal esm implementation

[MylesBorins]

Hey All,

Like everyone here I've been thinking about this a whole bunch. For a while I've been trying to come up with a minimal implementation that we can iterate on. Something that could offer a fast path to deflagging.

Features

• esm works with flag --experimental-modules
• must use .mjs as entry point for node binary
• no transparent interoperability
• import.meta.require for cjs interop in esm
• dynamic import for esm interop in cjs
• package-name-map compliance for esm specifier resolution
  • must provide full path to module
  • no support for importing directories

Try it out today

Repo: https://github.com/MylesBorins/node/tree/esm-kernel

Download: https://nodejs.org/download/test/v11.0.0-test2018070976df5841a1/

$ NVM_NODEJS_ORG_MIRROR=https://nodejs.org/download/test   nvm install v11.0.0-test2018070976df5841a1

warning super naive implementation, lots of room for improvement... but it shows off the desired UX and all the tests pass locally on my machine.

Upstream PRs

• import.meta.require
  • Import.meta.require node#21317
• package-name-map compliance
  • modules: remove file extension and directory resolving for esm node#21729

TODO

• improve implementation
• get tests working on all platforms
• build consensus
• ensure loaders works
• ensure vm works

Note

I have not opened a PR upstream to remove transparent interoperability as I want to be respectful of the current conversations going on within the group. While import.meta.require and limiting extensions are not guaranteed to land, I do think that they are worth discussing independently on their own merits.

[WebReflection]

Quick review from user perspective.

Not only the binary

You wrote "must use .mjs as entry point for node binary" but I've noticed .mjs is mandatory as entry point regardless it's a binary file or not. Since there is a mechanism to start ESM, I'd love to have a flag that forces ESM as in node --experimental-modules --esm index.js.

Overall working ...

Overall everything seems to work as expected:

• you can import fs from 'fs';
• you can import rand from './module.js'; where module.js contains export default Math.random();
• you can const {require} = import.meta; and bring in CJS as you go
• you can require('./module.c.js') with module.exports = import('./module.js').then(module => module.default); in it and it works as expected, also same exact module

import fs from 'fs';
import rand from './module.js';

const {require} = import.meta;

const readFile = require('util').promisify(fs.readFile);

Promise.all([
  readFile('./package-map.json'),
  require('./module.c.js')
]).then(([data, crand]) => {
  console.log(rand === crand);
  console.log(data.toString());
});

... but !

I am not sure I am doing it right with the package-map.json file, but I've used this content:

{
  "path_prefix": "./node_modules",
  "packages": {
    "flatted": { "main": "esm/index.js" }
  }
}

and there's no way I can import flatted from the previous index.mjs file via import {parse} from 'flatted';

The ./node_modules/flatted/esm/index.js is a 100% valid ESM file already working on browsers and also NodeJS (but you need to use CJS otherwise it fails).

Last, but not least

• I was incapable to bootstrap ESM via node --experimental-modules -e 'import("./index.js").catch(console.error)'
• I was unable to use __filename or __dirname and the import.meta wouldn't provide any alternative

Wishes

• the .mjs at this point is needed just as entry point, meaning it's not really needed if not to signal how an env should be parsed. I wish we could use a flag for that instead of an extension.
• any alternative to __dirname or __filename would be great. I know it's relatively trivial to have both on user land but the following seems unnecessary boilerplate

import path from 'path';
const {
  pathname: __filename,
  __dirname = path.dirname(__filename)
} = new URL(import.meta.url);

console.log(__filename);
console.log(__dirname);

[bmeck]

Can you define "transparent" interop here so that it doesn't get confused.

I remain strongly opposed to import.meta.require in light of my discussions on the related PR.

I would like to better understand the reasoning behind all of these choices rather than just seeing something that works. It would be nice to see a write up on what things were chosen not to be supported and why the use cases they provide can be served with this minimal implementation.

• I have a particular concern about not being able to import CJS/C++ that I have brought up before and consider it necessary for any minimal implementation due to ordering concerns. Explanation of how those concerns are alleviated still in this minimal approach would be good.

• This seems to imply a Major breaking change of requiring the entrypoint to be ESM once unflagged? How does this affect "bin" scripts that currently execute as CJS.

• The package-name-map compliance seems to be at odds with current discussions about migration patterns in Patterns for interoperability #139 . It would be good to know how migration is expected to occur here.

[demurgos]

The package-name-map compliance seems to be at odds with current discussions about migration patterns in #139 . It would be good to know how migration is expected to occur here.

To clarify, the issue with the package-name maps is that it requires the specifier to have an extension. Unless there's is another way to branch on the module type used by the consumer, this rules out the patterns relying on "mjs + js" (see table in #139). This currently leaves only "Default Export" and its clunky API as a completely safe pattern.

[hybrist]

I really like the direction of going for a more minimal solution.

I agree with @bmeck that there's some open questions. But I'd like to address them from a perspective of "what is the use case". So "Loading of polyfill or APM instrumentation that is implemented in CJS" instead of just "ordering concerns". Because that will most likely inform how we talk about it. When refactoring an application with "random" CJS files, the order is not as problematic in the general case.

[bmeck]

@jkrems we have files for configuration and singleton modules at work that need to evaluate prior to others. They configure things like servers and database connection pools. These are outside of polyfill or APM concerns.

[hybrist]

But I assume (hope) that they don't set random globals that other files then just assume are initialized..? If they do, I would treat them as "polyfills" (mutate global state to expose additional APIs).

[hybrist]

For additional color, the following will "just work":

import fs from 'fs';
import randomLib from 'esm-lib';

const db = import.meta.require('db');

db.accessTable(); // db init will still happen before this

[devsnek]

for even more color, the following will "just fail":

import.meta.require('x_polyfill'); // only available as cjs
import "something_expecting_x";   // only available as esm

I really don't like the term "just works" because it leaves out all the things that don't "just work"

[hybrist]

One possible answer for "how would polyfills work in that world" would be "it requires an intermediate module", e.g.:

import.meta.require('polyfill-a');

await import('./run-app');

But that definitely reduce the abilities of modules that need to ensure a polyfill (no static import of actual code, pretty awkward exports).