The dreaded CommonJS (CJS) to EcmaScript Modules (ESM) conversion. As the Javascript ecosystem continues to evolve and we continue to standardize, one unfortunate side-product for us NodeJS users, is the conversion to ESM.
Admitted, as most things Node, since it isn’t running in the browser context we don’t have to convert to ESM, and it likely won’t be deprecated anytime soon.
But hey, what’s the point in standardizing if we’re not going to follow it?
Some idealistic version of me – and perhaps you – made the decision to convert our relatively large NodeJS codebase from CJS to ESM.
Honestly?
I can’t say I recommend it. Outside a few quality of life improvements, and knowing I’ve committed to a more recent standard of doing things, I’m not yet sure its worth it.
This article is very long – much longer than I anticipated it would be. As such, I hope its something you can come back to and read again and again as you uncover more problems while progressing in your own journey with converting CJS to ESM.
And if you’re really short on time and just wanna get the thing working, hopefully my Cheat sheet at the very end of the article can help you get there.
This PR had a staggering ~10.000 lines changed! Sure, most of them were the same line that was slightly changed, but that’s a lot of going through code!
But, since I went through the trouble already and I see more and more developers being curious about exploring this exercise, let me pass on a few of my experiences to you.
To structure those experiences, I’ll first start with how I went through everything, the challenges at each step and how I overcame them.
At the end, I will try to provide a cheat sheet of symptoms and their solutions.
So if you’re really set on making this change, or just curious on mine, read on.
Getting started: Migrating Requires to Imports
The first thing that needs doing when migrating from CJS to ESM is to go to package.json and add "type": "module", to tell the application its using ESM. Making just this one change and trying to run your app the same way you usually do, will most likely break it.
Yeah, that’s all it takes.
Thankfully the solution to fix this, is simply to go through the entire application and change all your requires or imports. Hah! Simple huh?
That fear creeping up in the back of your mind, that’s you realizing that changing all your requires/imports, is going to take a while.
While I let you simmer on the scope of that task for a bit, before we dive into fixing it, let’s proceed to set up tsconfig.json so we get the appropriate Typescript support during this conversion.
In our case, the relevant changes were:
"target": "ESNext",
"module": "NodeNext",
"lib": ["ESNext"],
"isolatedModules": true,
"noImplicitAny": true,
"strictPropertyInitialization": true,
"noImplicitThis": true,
"moduleResolution": "Node16",
"sourceRoot": "/"Sentry recommended we change sourceRoot to just / instead of ./, with the following note:
“Set sourceRoot to “/” to strip the build path prefix from generated source code references. This improves issue grouping in Sentry.”
I thought that was pretty cool thing to mention.
Now, an additional side-effect from these changes add some frustration on top of the requires/imports, as we now also have to append them all with .js.
Alright, hopefully you got distracted from that initial fear and you have calmed back down. Naturally, I’m here to provide you with an anxiety-reducing antidote for how to do that, not to create an anxiety-inducing task ahead of you! 🙂
So what I found to be incredibly helpful was this handy little regex we can use directly within VSCode to search the codebase for all imports/requires:
(const|var)(\s.*\s*)=\s*require\(((‘|”).*(‘|”))\)(;?)$I can’t take credit for it, I found it on Medium, posted by “Airy” on the same topic, that has also proven incredibly useful.
It takes a bit of reading and understanding regex, but ultimately it just looks for a very specific pattern with any number of symbols and letters in between const or var and require.
If you’re using import already, instead of require, you would use the following regex to match them:
import (type )?\{?[^}]*\}? from "([^"]+)";We can follow that same pattern to append the .js to all our imports! Yay! Antidote!
Well I am here to make it easier for you 🙂
Assuming you have converted all your requires to imports (if not, do that first), the last step is to add the .js to them:
import $1\{$2\} from "$3.js";All done!
The expression is not constructable
After getting imports and their file endings in place and fixing the tsconfig, we should theoretically be ready to run the project with ESM.
However, a few of the libraries we use in our codebase, suddenly started giving the above error message:This expression is not constructable.
This would typically be libraries that in some way, expect you to generate a new class instance, ie new MagicLoginStrategy (using passport-magic-login).
After a bit of googling and some helpful comments on Discord, I found a solution that seemed to work well for us: simply adding .default to instantiating and that’s it.
In my case, I made the following change:
// BEFORE
export const magicLoginStrategy = new MagicLoginStrategy({ ...
// AFTER
export const magicLoginStrategy = new MagicLoginStrategy.default({ ...Simple.
Now, thanks to the help insights from users Bert and Jon from the TypeScript Community Discord, I was able to dig up this next, very fancy, trick.
We can use the tool arethetypeswrong, to check how types are wrong for any given library.
In my case of using passport-magic-login, we can see the library incorrectly exports the default for Typescript.
CSM Changes the default import
As explained on the website, the types contain export default which TS can’t interpret correctly.
Because the landscape is currently supporting ESM and CJS, and it isn’t exactly in a perfect state, sometimes export gets muddied.
As far as I understand, if the original library is written in CJS and we import that from an ESM project, it changes the default import, from the actual default export, to an object that contains the default property.
In other wordsimport foo from 'foo
becomes
import fooModule from 'foo'
const foo = fooModule.defaultA brilliant fix
While I haven’t personally tested this fix out, it may work with some configurations of tsconfig and imports.
If you’re seeing this error, you could attempt to solve it with a // @ts-expect-error comment annotation. Not only does this quelch typescript complaining, but if a newer version of the library ever changes the export, you’ll be notified (because it would no longer be throwing an error), thus prompting you to come back and remove the annotation.
I did not have any luck with my particular use of libraries and the tsconfig settings I made, but I certainly think this trick is worth a try, as its likely an even better solution than adding .default.
Users @Bert and @Jon from the TypeScript Community discord, thank you both for these insights! 🙏
Fixing dotenv
As soon as I had everything up to this point in place, I started getting weird error messages when starting the development server, that process.env couldn’t be found or certain environment variables weren’t present.
Given the changes I made up until this point I found this very weird.
Thankfully my time with Javascript and the way it works has taught me a thing or two about its event loop and the way it handles and processes modules, plus how this is different between CJS and ESM.
CJS will “require” a file (whether you use import or require syntax, its still requiring under the hood) and can have it linger in memory until it needs to run any code within this file.
ESM works a little differently in that it immediately imports every file, all of its imports, and all of those imports and so on and so forth until it has loaded and evaluated all code in the application.
Here’s a great article on the differences.
This made me realize that what was going on under the hood, is that I was trying to use an environment variable before it was “instantiated”. Yes, this can happen in Javascript.
Well, depending on how you have your setup.
But if you, like me, just like to keep it simple and use the dotenv package, this could be a headache of yours too.
The solution to this issue is simply to pull your dotenv import to the top of your list of imports in your main app file.
Eh, and just cause things are never easy, specifically this package needs you to resolve this in ESM, with a nested import, so it looks like this:
import "dotenv/config.js";Easy!
Changing my development toolset
I think I’ve been pretty vocal about my preference for certain tools over others, one area I have been particularly nitpicky about has been what to use to run my local development server.
After an unhealthy amount of mourning I had to put ts-node-dev in the grave. It simply isn’t built to work with ESM.
However, given that ts-node-dev is “just” a tool built on top of ts-node the choice becomes obvious: We can simplify a bit, and use the more stable tool ts-node to run code locally.
While I have been extremely happy with ts-node-dev and its performance, I am noticing a stability after a few months of usage with ts-node that I just didn’t have previously.
Not having random crashes or server reloads not being triggered, it’s a nice change of pace being completely able to rely on my development tools.
Well, at least so far.
ts-node can be used with node imports (previously called ‘loaders‘) so I wrote a simple one. Well, I actually borrowed one from the wonderful AdonisJS project, as they came up with this simple import script.
It doesn’t do much but it does ensure that files are loaded in the desirable order.
tsnode.esm.js
/*
|--------------------------------------------------------------------------
| TS-Node ESM hook
|--------------------------------------------------------------------------
|
| Importing this file before any other file will allow you to run TypeScript
| code directly using TS-Node + SWC. For example
|
| node --import="./tsnode.esm.js" bin/test.ts
| node --import="./tsnode.esm.js" index.ts
|
|
| Why not use "--loader=ts-node/esm"?
| Because, loaders have been deprecated.
*/
import { register } from "node:module";
register("ts-node/esm", import.meta.url);I’ll be honest, I couldn’t tell you exactly why this bit is important, but I know that it did fix my ts-node and I was finally able to boot up my application again 🙏
Amazing!
Using Speedy Web Compiler (SWC)
SWC is an incredibly popular tool in the ecosystem, and is slowly becoming adopted by many frameworks and tools as the fast way to transpile Typescript.
I don’t see no problem with using this, in fact I would have loved to, and even played around with doing so.
It plays incredibly well with ts-node too, so it really simplifies the configuration so that all you have to do is first install it, and then add the following to your tsconfig.json:
{
"ts-node": {
"swc": true
}
}This is well documented on ts-nodes own website.
However, I did not personally have any success getting this to work, because using SWC effectively enables transpileOnly, which in turn prevents the use of emitdecoratormetadata.
Given the latter is a requirement in our codebase due to some of the libraries we use (typegoose), I have had to delay the use of SWC.
It was possible for me to use it for pretends sake, and for the few iterations I played around with it I can say its much faster than using just ts-node.
Unfortunately, using it as a permanent solution in our repository is not an option due to the constraints mentioned above, but I dare dream one day it will be.
Now, just as my terminal was finally back to giving some familiar and expected outcomes, I noticed redis was acting up…
Redis being a trouble maker
Redis and I usually play well together and have had few squabbles in the past. At this point I was pretty pessimistic about Redis suddenly throwing a fit, seemingly out of nowhere.
In short, our development environment has always have redis print the message “Connected to Redis”, when connected with no issues.
After the above change in tools, this was no longer happening and instead I saw multiple instances of “Redis is already connected or trying to connect”, mixed in with my usual “Connected to Redis” message, apparently with no rhyme or reason to it.
The quick reader may have already uncovered what easily took me a few extra hours.
Redis was suffering from the same problem that dotenv had. Given the changes in how ESM works, I wasn’t instantiating Redis in a timely fashion for it to be ready when connections were attempted.
I actually ended up isolating everything redis in it’s own file redis.js and just did an incredibly simple initialization there:
import { Redis } from "ioredis";
const redisClient = new Redis(
process.env.REDIS_URL || "redis://localhost:6379",
);
// Ensure the client connects and handles errors
redisClient.on("connect", () => console.log("Connected to Redis"));
redisClient.on("error", (error) => console.error("Redis Client Error", error));
export { redisClient };This was a welcome change, as it massively simplified the old file in which the initiialization took place, muddled up with the database instantiating itself too.
The last bit here, is pulling the redis.js import to the very top of our app.ts file, just below the dotenv/config.js import. Ét voilá, it now works!
I figured it had to go to the top, because – again – of the way ESM loads modules. Many other files and classes in our project were dependent on Redis being started, so obviously pulling it to the top of the entire app, simply solved that issue.
Why I think Node subpath imports are not ready
At this point my big update was more or less ready to ship and most of the issues I had seen were already fixed. I still had Sentry working, Redis running and a local development toolset that seemed adequate. Great!
However, one does tend to get caught up in optimization, and one such optimization I was hoping to make, was to make use of Node “subpath imports”. On paper, they’re amazing. A convenient way to tell your editor how to read and set imports, all with a neat prefix #, and you can easily start pointing to files.
I ended up changing all of my imports to use node subpath imports, and development immediately broke:Error [ERR_MODULE_NOT_FOUND]: Cannot find module
ts-node couldn’t find the modules, when using subpath imports. However, using something like nodemon and pointing directly to the compiled output folder seems to work fine.
Now, my reason for thinking they aren’t exactly ready, is that a lot of projects use ts-node or an equivalent tool. And rightfully so. I believe they are better tools than having to rely on nodemon while offhanding compilation to another tool, hoping the two play nicely together.
Technically, subpath imports work just fine, but they don’t play nice with typescript. When a file is specified, say in #src/common/interfaces/user.interface.ts, the subpaths – or Typescript, I don’t know honestly – aren’t smart enough to convert that to point it to the output folder. I mean, the compiled output.
Obviously node can’t run .ts files, and ts-node can’t find it because it’s looking for an absolute path, not a relative one.
Since this doesn’t fly better with Typescript as of today, I’m holding off on implementing it until it works better, or I start working with a simpler project and either can work my way around it or choose other tooling 🙂
Rounding up
At the end of the date, changing to ESM isn’t that big of a headache, looking in hindsight. Most of the time I spent, was fixing issues I couldn’t immediately figure out why was happening, because they were happening as side-effects to the changes I was introducing.
Early in the article I stated, that I wouldn’t recommend this change.
While this is still my stance overall, knowing what I know now, I have softened up a bit. Armed with the experience and knowledge of understanding the moving parts that go into such a conversion, I don’t find it quite as intimidating or troublesome to go over again.
But I recognize that even if the obligatory changes, most of the time ends up being spent on all the side-effects and libraries that needs massaging or even replacing.
But I also want to highlight some of the good parts, as I’ve really come to enjoy those.
Better development environment
The first benefit, is that our local development environment grew incredibly stable, almost immediately.
Normally, if the local server was running for too long, or a change was too big, a new file was introduced, or a number of other odd activities happened, the server would crash.
It wouldn’t throw an error, but just sort of fall out of sync with the changes happening in the environment, which required a shutdown and starting it again.
Ocassionally that would be a forced shutdown, killing the process, and then restarting.
I can’t stress enough how amazing it is to have a stable environment that just never seems to crash.
On top of that, it seems that rebuilds of the local development are happening faster than they used to with ts-node-dev.
That may be recency bias talking, because I certainly remember choosing the above, because it was by far the fastest out of the options I tested at the time.
Nontheless I’ve put down a note that it was faster, so lets leave it at that.
A sidenote here is that it may be possible to use the much faster tsx, but I haven’t gotten around to experiment with it.
I’d be super curious to hear your results with it, so don’t be afraid to drop a comment all the way at the bottom 👇
These benefits alone are rapidly growing on me, and almost seems worth the conversion by themselves.
But that’s not all that happened!
Less tooling
One way I love to work on and improve codebases, is by reducing complexity. That’s a broad statement, but one way it comes into form is by reducing tooling.
Fewer tools, means fewer things that play together, means fewer things that can be wrong.
Ultimately, fewer moving parts, is a guiding principle in how I interpret “reducing complexity”, and removing tools totally fits with that memo.
Faster Deploys
Given our application is a separated backend, with a separated frontend, we’ve always had to maintain two applications that need to work together. Two sets of types (lots of duplication), two pipelines, two deploy systems, two everything.
Something I was not expecting, was our deploy to speed way up.
Previously, a deploy, the last step in our CD pipeline, took around 3-5 minutes. After the change to ESM, it consistently finishes within 1-1.5 minutes.
That’s over 100% improvement in speed!
I don’t exactly know why this happened if I’m being truthful, but I suspect it has to do with ESM enabling a lot more tree-shaking than CJS.
My best guess would be that’s capable of stripping a lot of the code, quicker than CJS would, because it imports all modules right away, and does treeshaking appropriately.
My progression notes
Heres a gist I used to work through the entire thing, while figuring out what to do:
https://gist.github.com/aarhusgregersen/54fdf463b7e000e64365cc4bd096f67d
It includes some of the relevant files, plus notes on what things I tested, and their results, it might be worth a read and could be insightful, if you’re setting out on this venture yourself.
Do note that the files in the gist are not a reflection of the final result, but rather just copies at some point in time during my conversion time, as I needed to more thoroughly document things in order for myself to understand each part.
Sources and recommendations
Before we get to the cheat sheet, I just wanted to include the sources I stumbled upon, as I went through this process and all this research.
First, this excellent gist on how to convert your project to ESM, by Sindre Sorhus, was incredibly helpful. In a convenient and easy-to-understand manner, it lays out the steps required to perform the migration.
This example package.json file includes some build-scripts and libraries that I hadn’t thought of using, that came in handy mid-way through the migration.
I experimented with concurrently and while it does work great I found that pnpm can do a similar thing, but more importantly I wanted to reduce tooling, not add extras. So I went with the most straightforward path, which was ts-node-dev -> ts-node.
I also found great insights searching the TypeScript Community discord, and both posting my own issues, but also exploring others, when I came across similar errors.
Recommendations
Take the time to do one step at a time, and test it works for both building production and running development.
A lot of the pain I had during this process, was not having a structured outline for how to approach, as I have now that I’ve been through it.
I would have been much better off doing one thing and then the next, continously finishing one before moving on.
Ie. Redis first, then dotenv, then tsconfig.json.
Incremental changes are the way to go!
Take it slow and read error messages
A lot of the issues that made me “hit a wall”, and stop wondering, couldve most likely been easily solved with a bit of googling and researching.
Fundamentally CJS and ESM are different, but not so different that most developers can’t understand it in a solid 20 minutes or so. Obviously, not the full spectrum, but enough to spot some differences.
Google errors as they come up. Make slow incremental progress and overall you’ll be hitting that goalline much sooner!
Make every new project ESM from the get-go
While this article has been exclusively about the conversion progress I want to solidify that I am certainly for ESM. I think its a much better experience overall. But for larger existing projects, unless they absolutely have to convert, I can see the pain being too large to be worth it.
As such, if you’re starting new projects, make absolutely sure you make them ESM from the beginning. Lets do ours to convert this ecosystem 🙂
Cheat Sheet
| Issue | Solution |
This expression is not Constructable | Append a .default on the class you are trying to instantiate |
| Module not found | Make sure your tsconfig.json is in order. Append .js to import statements |
process.env not found | Move import "dotenv/config.js"; to the very top of your first app file. |
Connected to Redis and cant connect to redis: already connected | Move redis import right under dotenv import. |
Best of luck!