In a previous post, I talked about the layout of DefinitelyTyped and how it was indeed a monorepo, albeit a funky one. In short, packages were laid out (more or less) like this:

types/
  gensync/
    tsconfig.json
    index.d.ts
  node/
    tsconfig.json
    index.d.ts
    package.json
  react/
    tsconfig.json
    index.d.ts
    package.json

And so on. Each tsconfig.json file contained bits like:

{
    "compilerOptions": {
        "baseUrl": "../",
        "typeRoots": [
            "../"
        ],
        "types": []
    }
}

This config means that when a types package looks for itself or another package, it can map to other directories in types; no package.json is needed. At publish time, we detected dependencies and add them. If a package needed an external dependency, then the package would need a package.json file with that dependency declared.

This provided a monorepo-like feel without any symlinking, but with many downsides, including:

• Long npm install times when external dependencies are needed, especially when testing the entire repo. The tooling just looped over every folder with a package.json and ran npm install.
• Completely unrealistic module resolution (no node16 / nodenext, no export maps, etc.) thanks to the use of baseUrl, typeRoots, and paths. Not even typesVersions works.

I also talked about what we could do to remedy the situation, which boils down to “what if we were just a monorepo like everyone else uses in the JS ecosystem and let a package manager handle things”?

Making fetch happen

Obviously, all of that was a good 6 months ago. There were some unresolved blockers that made me put the project on the backburner. What changed?

Recently, Andrew merged fetch support into @types/node. Yay!

But, you might have noticed that only @types/node@20 got this feature. Surprise! It’s the second bullet point from above. DefinitelyTyped’s fake module resolution broke resolution inside undici-types, the package @types/node depends on to provide fetch types (without depending on undici itself, which is vendored into Node). The effect of this is that we could only add fetch to the latest types for Node, not to any older versions. In fact, if @types/node@22 were needed, we’d have to drop it from @types/node@20! Boo.

This problem shuffled the whole “DefinitelyTyped monorepo” thing straight to the top of our interest list.

And so with that, I’m happy to say that after a few weeks of effort from Nathan, Andrew, and myself, we’re actually doing it! DefinitelyTyped is becoming a monorepo!

Hello, pnpm

If you’ve read the previous posts, you won’t be surprised to find that we’re using pnpm to do this. All modern package managers have some sort of monorepo support these days, but DefinitelyTyped’s unique situation limits what we can use. Specifically, DefinitelyTyped contains multiple versions of the same package. For example, we currently have @types/react v15-v18, @types/node v16-v20, and so on. Both npm and yarn exit early when they see two workspace packages with the same name. Understandable!¹ But, we need to do it somehow.

With pnpm, this “just works”. Internally within pnpm, workspace packages are identified by their paths, so there’s no conflict. Then, when pnpm goes to resolve packages, it only cares about the name and version. This actually means we get something better than just “it doesn’t fail”; it can actually resolve to these workspace packages based on their versions! It behaves just as though the packages were provided by the npm registry. So long, paths.

There’s a bunch more goodness pnpm provides, but for now, let’s just look at the new layout.

The new layout

Anyone who’s worked in a monorepo will not be surprised by the new layout. Now, we have:

types/
  gensync/
    tsconfig.json
    index.d.ts
    package.json # new!
  node/
    tsconfig.json
    index.d.ts
    package.json
  react/
    tsconfig.json
    index.d.ts
    package.json

Every @types package now requires a package.json, even if it doesn’t have any external dependencies. Let’s take a look at what’s inside. Here’s the new bits of package.json for @types/jsdom:

{
    "private": true,
    "name": "@types/jsdom",
    "version": "21.1.9999",
    "projects": [
        "https://github.com/jsdom/jsdom"
    ],
    "minimumTypeScriptVersion": "4.5",
    "dependencies": {
        "@types/node": "*",
        "@types/tough-cookie": "*",
        "parse5": "^7.0.0"
    },
    "devDependencies": {
        "@types/jsdom": "workspace:."
    },
    "owners": [
        { "name": "Leonard Thieu", "githubUsername": "leonard-thieu" },
        { "name": "Johan Palmfjord", "githubUsername": "palmfjord" },
        { "name": "ExE Boss", "githubUsername": "ExE-Boss" }
    ]
}

That’s a lot of stuff. Much of this is information that was previously a part of the index.d.ts “header”, i.e. something like:

// Type definitions for jsdom 21.1
// Project: https://github.com/jsdom/jsdom
// Definitions by: Leonard Thieu <https://github.com/leonard-thieu>
//                 Johan Palmfjord <https://github.com/palmfjord>
//                 ExE Boss <https://github.com/ExE-Boss>
// Definitions: https://github.com/DefinitelyTyped/DefinitelyTyped
// Minimum TypeScript Version: 4.5

In the new layout, we’re going to be using a package manager, so we need to declare a name and version to get packages to link up. That’s the first bit of the header. At that point, we may as well just move everything into JSON and be done with it. Additionally, this means that tools wanting to grab info about a DefinitelyTyped package don’t need to parse the header text; it’s all in package.json.

Let’s break down the fields.

private

{
    "private": true
}

This is always set to true, telling pnpm to not attempt to publish this package to the registry. The DefinitelyTyped publisher handles publishing. Packages that had a package.json previously already had this set, so this is nothing new.

name

{
    "name": "@types/jsdom"
}

This is new! Previously this was declared only via the directory name, but now we’re going to be using pnpm to handle things, so we need to specify this.

version

{
    "version": "21.1.9999"
}

This one’s funky; it’s almost what we put in the header, but with a patch version of 9999. When DT packages are published (automatically on a schedule), the patch version is generated; it’s just whatever the previous version was, plus one. So the patch version that’s actually in the repo never matters.

At development time, we’re making use of the fact that pnpm can resolve to local versions. Normally, we could set prefer-workspace-packages, which would force pnpm to always link to the local workspace package. But we actually have a few packages which intentionally point to old versions of @types packages. If we were to do the much nicer thing of using 0 as our patch version, the version from the registry would always be chosen instead. So, we can’t use prefer-workspace-packages. Instead, we just pick an arbitrarily high patch version, such that it will always be newer than what’s in the registry, hence 9999. 9999 publishes ought to be enough for anyone, right?

projects

{
    "projects": [
        "https://github.com/jsdom/jsdom"
    ]
}

This is an array of helpful links to info about a project. Usually it contains a GitHub link, but can sometimes contain more.

minimumTypeScriptVersion

{
    "minimumTypeScriptVersion": "4.5"
}

This defines the minimum supported version of TypeScript version for a package.

dependencies

{
    "dependencies": {
        "@types/node": "*",
        "@types/tough-cookie": "*",
        "parse5": "^7.0.0"
    }
}

This isn’t new, but it is bigger! Dependencies on @types packages are now explicit. No longer can every package access every other package; pnpm won’t link them. This removes the complexity of the infrastructure; we don’t need to parse the code or rely on heuristics to figure out what packages depend on what.

Additionally, this makes pnpm fully aware of how the packages interrelate, meaning that we can use fun features like --filter (more on that later).

devDependencies

{
    "devDependencies": {
        "@types/jsdom": "workspace:."
    }
}

This is new. For the most part, this will contain just one thing; a self-dependency. Without the baseUrl / typeRoots / paths combo, a package can’t find itself anymore, but that’s the API that we’re wanting to test. pnpm doesn’t yet support creating self-links, so we do it ourselves using a workspace:. specifier.²

This list can also contain packages that are needed for testing. This technically an improvement over the previous setup, which didn’t allow devDependencies at all. But, it’s generally better to not have any testing dependencies anyhow.

owners

{
    "owners": [
        { "name": "Leonard Thieu", "githubUsername": "leonard-thieu" },
        { "name": "Johan Palmfjord", "githubUsername": "palmfjord" },
        { "name": "ExE Boss", "githubUsername": "ExE-Boss" }
    ]
}

This is a list of the users that “own” the package. They get pings when people send PRs to packages and can approve them. This used to be in the as URLs (as that’s the syntax needed for the contributors array in package.json), but our tooling only wants usernames and loads of people incorrectly typed their GitHub profile URLs. For owners that aren’t directly on github, url can still be passed (though not shown above).

nonNpm, nonNpmDescription

{
    "nonNpm": true,
    "nonNpmDescription": "Google Maps JavaScript API"
}

My example didn’t have these two, but some of the packages in DefinitelyTyped describe things that aren’t npm packages at all. For example, @types/google.maps describes the Google Maps API (a global), and had a header like:

// Type definitions for non-npm package Google Maps JavaScript API 3.54

This info is used to inform various checks and is carried into the published package. In the new layout, this information is represented in JSON.³

Installing dependencies

Let’s start by doing the naive thing and just run pnpm install in the root of the repo.

$ pnpm install
Scope: all 9114 workspace projects
...
Done in 3m 35.4s

Wow, that’s a lot of install. But it’s a major improvement over the previous layout, where installing the entire repo (with 10x fewer package.json files) took some 30 minutes.

The good news is that those working on DT don’t actually need to install the entire repo. pnpm supports filtering. Let’s say I’m working on @types/node, and want to be able to test it and any packages it depends on. I can run:

$ pnpm install -w --filter '...@types/node...'
Scope: 2722 of 9114 workspace projects
...
Done in 1m 2.5s

That’s a good bit better! Since we have explicit dependencies, pnpm can actually figure out what packages are needed for @types/node and install those, but also figure out which packages depend on @types/node and install those too. The -w tells pnpm to also install the workspace root, which is needed to get the DefinitelyTyped tooling, linters, dprint, etc.

What about package that isn’t so hefty?

$ pnpm install -w --filter '...@types/lodash...'
Scope: 372 of 9114 workspace projects
...
Done in 7.2s

Now we’re talking. Most packages won’t need to do a huge install, so long as people read the docs (🙃) to know how to avoid the big install.

From this point on, the workflow is the same as DefinitelyTyped was before.

Filtering in CI

There’s one other cool trick that we can use in CI; not only can we filter by package name, but we can also filter by what changed since a specific git ref. In a PR build, we can use:

$ pnpm install -w --filter '...[origin/master]'

And only get what we need.

Other misc improvements

There’s also a grab bag of other improvements that come with this change. In no particular order:

• Having to redo a bunch of the DefinitelyTyped tooling has led to improvements like dtslint-runner no longer bailing early on certain kinds of errors. Many more things are collected for reporting at the end such that doing one thing wrong doesn’t hide the problem until a second run.
• As a part of making everything work in the new monorepo, we manually fixed a few hundred packages. These packages were silently broken (or at least, weird) in various ways. For example, multiple packages imported the events library. This could mean @types/node, but it could also mean @types/events. In practice, it resolved to the latter, but then sometimes, the tooling would say the package depended on neither (probably due to a bug in the implicit dependency resolution). Now, each package actually has to say which they need. There are other weird things besides just this; invalid references directives, packages depending on the wrong versions of things, etc.
• Having a complete working package.json for every package means that one can theoretically just npm pack and get a working tarball. This is likely to become useful for tools like Are The Types Wrong, although the publisher still does a bunch of stuff (notably, deciding which files actually get included in each package, which is still mostly implicit).
• It turns out that a load of react-based types packages are broken at the moment due to @types/react using typesVersions. Since typesVersions is in package.json, but baseUrl / typesRoot / paths skip package.json resolution, packages that depend on @types/react always get the types meant for TS 5.1 and above. Oops. With actual node_modules linking, this isn’t a problem and things work as intended. Another reason to speed this along.

It’s not all rainbows and sunshine

Everything I’ve described so far has been an improvement over the previous layout. But there are some warts left to figure out.

Using shared-workspace-lockfile=false

The astute reader may have noticed that the performance of pnpm install seems way slower than expected, especially given the numbers I achieved in the previous post about making pnpm faster.

The reason for this slowdown is our use of shared-workspace-lockfile=false. This is a lesser-used option which instructs pnpm to instead handle each workspace package individually. Each package gets is own dependency graph, and calculating that 9000 some times is a lot slower than doing it once.

Why enable it, then? It’s a bit hard to fully explain. I filed an issue for it upstream, but the gist is that since pnpm is stricter about how it handles dependencies (they aren’t just all hoisted to the top), the combo of so many packages forgetting to declare dependencies on @types packages along with some packages (outside DT) explicitly depending on @types/node@17 (why??) causes the “fallback” @types/node to point to that awkward v17 version.

I haven’t quite figured out what the best solution is; it’s possible that pnpm could gain a yarn-like hoisting-limit system to avoid this problem, or always resolve “fallback” dependencies to the latest version.

I had previously skipped working on this project due to this problem, but the upsides of the migration (especially with fetch getting thrown in the mix) tipped the scales. Although shared-workspace-lockfile=false is quite a bit slower, it’s still an improvement when installing the entire repo, and filtering provides a very straightforward way to reduce the cost of package installs.

Just to show what the difference is, here’s the install without this setting.

$ pnpm install
Scope: all 9114 workspace projects
...
Done in 1m 5.8s

$ pnpm install -w --filter '...@types/node...'
Scope: 2722 of 9114 workspace projects
...
Done in 30.2s

$ pnpm install -w --filter '...@types/lodash...'
Scope: 372 of 9114 workspace projects
...
Done in 7.9s

Not helpful super for a small number of packages, but quite a bit faster if you ever need to work with the whole thing.

git clean is broken on Windows

pnpm uses symlinks under the hood. On POSIX-ish platforms like Linux and macOS, this is all good; symlinks work for any user and behave as expected. On Windows, however, the story is different. For a very long time, the only way to get “real” symlinks was to gain elevated permissions. Without being an admin, the best you could hope for was a “junction”. I’m not sure I could explain the ins and outs of junctions other than to say that they’re like a symlink, but only for directories, and they act kinda weird sometimes. But, they do the job.

There’s a gotcha; if you run git clean, git treats junctions as directories! This is normally not a big deal, but every one of our packages has a self link, which makes the symlinks recursive. And since git doesn’t treat junctions like symlinks, git clean will just keep recursing infinitely until it hits the max path length. It may eventually finish, but without loads of errors and complaining.

There are two ways forward:

• Make git treat junctions as symlinks. I sent a PR for this earlier this year, but it hasn’t yet been accepted. I personally think this is the correct solution; if git clean had been implemented in shell scripts (like much of git), it would have treated junctions as symlinks and just worked. But git clean is written in C, so instead of going through git-bash, it goes through the shims which translate the POSIX-y file system accesses into the Windows API, and those shims disagree with git-bash.⁴
• Make pnpm use real symlinks. You’re probably confused; didn’t I just say that you needed elevated privileges to do that? Normally, yes, but if you enable Developer Mode, any user is able to make symlinks! And, I fully suspect that most people developing on Windows have this enabled. You even need enabled to enable WSL. This is probably a good idea whether or not git changes; real symlinks don’t have the same warts as junctions. I’ll disclaim that I haven’t actually proposed this change upstream. There are some gotchas in that it may be awkward to enable this automatically (what happens if you end up with a node_modules with both junctions and symlinks?), but I think it should be straightforward to detect.

For now, DefinitelyTyped has included a script Windows users can run to clean up node_modules; pnpm run clean-node-modules will find and delete all node_modules directories within the repo. Good enough for now.

Removing a package won’t expose breaking changes in newly-typed packages

This one’s subtle. Imagine we have a package @types/foo. Another package (in the repo or even external) depends on @types/foo. But, foo has just gained types, which means that it’s time to remove @types/foo from DefinitelyTyped.

In the old layout, we’d delete the directory and add it to notNeededPackages. When the PR that does this is merged, the publisher will publish one final version of @types/foo that contains only a package.json with a dependency on the real foo.

But, when you’re actually working on the PR that does this, the shim package hasn’t yet been published! If another package within DefinitelyTyped depends on @types/foo it will stop pointing to the one in the repo (it’s been deleted). In the old layout, things would stop compiling until dependencies are updated. But in the new layout, pnpm will just resolve to the latest version of @types/foo in the registry, which will be exactly the same code that is being deleted. This means that the PR will definitely pass CI, when it may actually fail later if the real upstream foo package has types which differ enough to break things.

There’s not really a great solution to this other than to ban external dependencies on @types packages that aren’t contained in the repo; that handles some of the situations but not all. (If you have any clever ideas, let me know.)

Removing a package isn’t reflected in pnpm git filter

In addition to the above, when we delete a package, pnpm’s behavior for --filter '...[origin/master]' doesn’t pick up on the removed package. In an ideal world, it’d see the package removed, and then figure out which local packages are affected by the removal. But, it doesn’t do that, either simply because the package is gone (so there’s no more edges to check), or due to the previous section (where the package is still there, just not in the repo). The workaround is to instead use pnpm ls --depth -1 --filter '...@types/removed...' to get some sort of list of what may need to be tested. In CI, if a package.json is removed from the repo, we also don’t use pnpm install --filter '...[origin/master]', resorting to a complete install.

Future work

After this is merged, we’re still not done! There is some exiting stuff that gets unlocked:

• Since there’s no more header, there’s nothing special about index.d.ts anymore. Since we’re trying to get DT as close to the upstream packages as possible, it may actually be wrong to have an index.d.ts if the package doesn’t contain index.js. We should be able to remove the requirement that packages have an index.d.ts.
• Right now, there’s no way to really verify that types work properly for people using nodenext resolution (though, some analysis has shown that DefinitelyTyped does better in this regard than those publishing types themselves).Now that everything works via node_modules, we could finally use enable those options in tsconfig.json and verify that things work. Maybe even required.
• We could offload even more of dtslint-runner onto pnpm scripts.
• We could move tests out into their own packages, forcing them to use the public APIs of the packages they’re testing. Or even, have multiple tsconfigs to make sure things work with various settings.

Anyway…

I hope this info dump was interesting; I’m really excited to see this change happen.

Big thanks to everyone involved with this big migration, including Nathan and Andrew from the TypeScript team, Zoltan of pnpm fame, and everyone who spent time finding more ways to speed pnpm and semver up for our ridiculous use case.

For more background, see the previous post about DefinitelyTyped.

TL;DR: DefinitelyTyped is huge; installing it in its entirety involves processing over 9,000 packages. And that’s slow! Or is it?

Taking a profile

Many people may not know this, but I’ve actually written more Go than I have TypeScript.¹ As such, when I have a performance problem, the tool I like to use is pprof.

More commonly, this tool is used when profiling Go, C, C++ code. And I like this tool! Lucky for me, there is a library which lets you use it with Node.² The API is pretty straightforward; you can start and stop both CPU and heap profiles, and write them to disk.

Unfortunately, that’s a little annoying, because effectively 100% of the time, I’m profiling a CLI application or someone else’s project where I don’t really want to inject the code. It does include some code to let you do node --require=pprof myScript.js, but there’s no way to configure its behavior.

So a few years ago, I made a little wrapper, pprof-it, which makes things much easier to use. You can check the README for more details, but in short, to get a pprof profile you just run:

$ pprof-it /path/to/script.js

pprof-it will start profiling both CPU and heap allocation immediately at startup then dump profiles to the current directory on exit. These files can then be loaded into pprof (or one of the many other tools which support the format, like flamegraph.com or speedscope).

So, let’s take a profile of pnpm install on one of my work-in-progress “DT as a monorepo” branches. (Forgive the roundabout way of running things; some of my fixes are already released, so I need to do a little movie magic.)

$ npx --package=pnpm@7.30.0 -c 'pprof-it $(which pnpm) install'

This actually OOMs on my laptop (I have yet to determine why), but on my desktop, I get this:

pprof-it: Starting profilers (heap, time)
    # a very long pause...
Scope: all 9031 workspace projects
    # a very very long warning about cycles (I need to file an issue for this!)
Lockfile is up to date, resolution step is skipped
Already up to date
    # another long pause
Done in 1m 39.7s
pprof-it: Stopping profilers
pprof-it: Writing heap profile to pprof-heap-286252.pb.gz
pprof-it: Writing time profile to pprof-time-286252.pb.gz

Great, now let’s run pprof:

$ pprof -http=: pprof-time-286252.pb.gz

Automatically, pprof starts up my browser and puts me right into the graph view. This view outside of Node profiles is very useful, but Node profiles have an unfortunate problem which leads to all anonymous (i.e. arrow) functions being counted as one node named “(anonymous)”.³ So, let’s flip into the flame view.

Already, I’m excited; this is every profiler’s dream. Two very obvious chunks of work attributed to real names I can search for. Roughly 50 seconds are spent in createPkgGraph and another 32 seconds in getRootPackagesToLink. I should note that at this point in my adventure, I know absolutely nothing about how pnpm works; I haven’t even checked out the repo. But, now I know exactly where to look! (If pnpm had been minified, I’d be in a much worse position.)

Working through the code

From the get-go I can see that there’s a lot of time spent in resolve. One thing I hadn’t mentioned was how I set up this huge monorepo; my initial version of the monorepo transition used version specifiers like workspace:../node to directly map packages to each other, avoiding the need for us to specify names/versions in every package.json (they’re already auto-generated by the DT publisher). Without even looking at the code, I (correctly) guessed that these paths were involved in the slowdown and filed an issue.

It turns out that this path mapping is actually a negative for other reasons as well, so I just rewrote my transform to use versions instead of paths. After switching to this new version, the profile looks like this:

Alright, that’s better already, down from ~100 seconds to 64 seconds. We’ll come back to resolve later.

createPkgGraph

The first block is the first “very long pause” (which happens even in the “new” version of the repo), so let’s start there. Searching the pnpm codebase, I find the offending function. It looks something like this (cut down for brevity):

function createPkgGraph(pkgs: Array<Package>) {
    const pkgMap = createPkgMap(pkgs);
    return mapValues((pkg) => ({
        dependencies: createNode(pkg),
        package: pkg,
    }), pkgMap);

    function createNode(pkg: Package): string[] {
        const dependencies = {
            ...pkg.manifest.devDependencies,
            ...pkg.manifest.optionalDependencies,
            ...pkg.manifest.dependencies,
        };

        return Object.entries(dependencies)
            .map(([depName, rawSpec]) => {
                const isWorkspaceSpec = rawSpec.startsWith("workspace:");
                const spec = npa.resolve(depName, rawSpec, pkg.dir);

                if (spec.type === "directory") {
                    const matchedPkg = Object.values(pkgMap).find((pkg) =>
                        path.relative(pkg.dir, spec.fetchSpec) === ""
                    );
                    return matchedPkg?.dir;
                }

                const pkgs = Object.values(pkgMap).filter((pkg) =>
                    pkg.manifest.name === depName
                );

                if (pkgs.length === 0) return "";

                const versions = pkgs.filter(({ manifest }) => manifest.version)
                    .map((pkg) => pkg.manifest.version) as string[];

                if (isWorkspaceSpec && versions.length === 0) {
                    const matchedPkg = pkgs.find((pkg) =>
                        pkg.manifest.name === depName
                    );
                    return matchedPkg!.dir;
                }

                if (versions.includes(rawSpec)) {
                    const matchedPkg = pkgs.find((pkg) =>
                        pkg.manifest.name === depName
                        && pkg.manifest.version === rawSpec
                    );
                    return matchedPkg!.dir;
                }

                // ...
            })
            .filter(Boolean);
    }
}

Alright, so we can sort of see what might be going on here. First off, we have pkgMap. By attaching to the code and looking at the variable, we find that it’s an object which consists of all 9,000+ packages. So doing anything with that is going to take a while.

At the top level, we’re already looping over every entry in the object via ramda’s mapValues. But, if we look inside createNode, we can see that it is also looping over all of pkgMap by calling Object.values(pkgMap)! This is quadratic; we’ll be doing 9,000 x 9,000 scans over the array. We could fix this by instead creating a mapping and accessing it instead. For example, one of the loops is just looking for all of the entries in pkgMap where pkg.manifest.name is some value. We could precalculate this mapping, producing an object of type Record<string, Package[]>.

The other loop is more complicated; this is where resolve comes in. We can see that we’re searching not for a specific name but for a specific set of packages whose paths map the one we specified (that workspace:../node from earlier). This one is tricky, but it’s possible that we could precalculate some table here too, depending on how sensitive this code is to path.resolve’s platform-specific semantics.

Speaking of precalculating… We just said that pkgMap was huge. But, for every call to createNode, we call Object.values(pkgMap)! The profile doesn’t explicitly state so, but this is really, really expensive. The good news is that pkgMap is never modified. This means that we could calculate this big array once and then reuse it, for example:

function createPkgGraph(pkgs: Array<Package>) {
    const pkgMap = createPkgMap(pkgs);
    const pkgMapValues = Object.values(pkgMap); // <-- NEW!
    return mapValues((pkg) => ({
        dependencies: createNode(pkg),
        package: pkg,
    }), pkgMap);

    function createNode(pkg: Package): string[] {
        // ...

        return Object.entries(dependencies)
            .map(([depName, rawSpec]) => {
                // ...

                if (spec.type === "directory") {
                    const matchedPkg = pkgMapValues.find((pkg) =>
                        path.relative(pkg.dir, spec.fetchSpec) === ""
                    );
                    return matchedPkg?.dir;
                }

                const pkgs = pkgMapValues.filter((pkg) =>
                    pkg.manifest.name === depName
                );

                // ...
            })
            .filter(Boolean);
    }
}

This turns out to save the bulk of the time. Yay!

Algorithmically, the code is still quadratic, but it’s still a lot faster and this kind of change is very safe, safe enough to be backported. I sent this one as a quick PR, and it’s now out in v7.30.4.

The fix to the quadratic-ness is going to be a different, more complicated change I plan to send later.

UPDATE: Later is now the past! All of the quadratic-ness has been fixed as of:

• perf(pkgs-graph): speed up createPkgGraph by using a table for manifest name lookup
• perf(pkgs-graph): speed up createPkgGraph when directory specifiers are present

getRootPackagesToLink

Let’s look at the second big chunk. Cut down for brevity again, we have:

async function getRootPackagesToLink(
    lockfile: Lockfile,
    opts: {/* some options */},
) {
    const importerManifestsByImporterId = {};
    for (const { id, manifest } of opts.projects) {
        importerManifestsByImporterId[id] = manifest;
    }

    const projectSnapshot = lockfile.importers[opts.importerId];
    const allDeps = {
        ...projectSnapshot.devDependencies,
        ...projectSnapshot.dependencies,
        ...projectSnapshot.optionalDependencies,
    };

    return (await Promise.all(
        Object.entries(allDeps)
            .map(async ([alias, ref]) => {
                // ...

                return {
                    // a bunch of props
                };
            }),
    ))
        .filter(Boolean) as LinkedDirectDep[];
}

Again, the profile is not being very specific. It’s just saying that a lot of time is being spent in getRootPackagesToLink. Thankfully, there’s not much code actually inside this function. It can only be the calculation of importerManifestsByImporterId, or the spread to produce allDeps.

I debugged this to try and get the size of these elements. getRootPackagesToLink is called for every package in the repo, and allDeps is small. So that’s not likely to be it.

The importerManifestsByImporterId loop, on the other hand, is suspicious. I just said that getRootPackagesToLink is called once per package in the repo. But, opts.projects is a big list of all packages in the repo! We’re quadratic again!

This is better than before, in theory; there are lookups inside the .map call below, but they’re efficient because they don’t loop over opts.projects (as opposed to createNode from earlier, which does do the linear lookup). But, getRootPackagesToLink is recreating this mapping every single time it’s called!

If we scroll down a little bit, we can find its sole caller:

const projectsToLink = Object.fromEntries(
    await Promise.all(
        projects.map(async ({ rootDir, id, modulesDir }) => [id, {
            dir: rootDir,
            modulesDir,
            dependencies: await getRootPackagesToLink(filteredLockfile, {
                // ...
                projects,
                // ...
            }),
        }]),
    ),
);

There’s that “for each package” thing again. Thankfully, we can again see that projects is not changing between calls. So, we can instead calculate this mapping once and pass it in to getRootPackagesToLink, again without changing much logic.

const importerManifestsByImporterId = {} as { [id: string]: ProjectManifest; };
for (const { id, manifest } of opts.projects) {
    importerManifestsByImporterId[id] = manifest;
}

const projectsToLink = Object.fromEntries(
    await Promise.all(
        projects.map(async ({ rootDir, id, modulesDir }) => [id, {
            dir: rootDir,
            modulesDir,
            dependencies: await getRootPackagesToLink(filteredLockfile, {
                // ...
                importerManifestsByImporterId,
                // ...
            }),
        }]),
    ),
);

Now drop the code to produce the mapping from getRootPackagesToLink and we’re done.

I sent this as a PR over too, and it also is available in v7.30.4.

The “final” result (for now)

Now that we have these two fixes in, let’s re-profile pnpm install for the newer version:

$ npx --package=pnpm@7.30.4 -c 'pprof-it $(which pnpm) install'
# ...
Done in 13.6s

Immediately, the difference is evident. There’s no longer a huge delay before I get the cycle warning. The whole thing now takes 13.6 seconds. That’s a huge improvement! It’s outlandishly good to be processing 9,000+ packages in such a short time.

What about the profile, though?

Much different. We can see that the huge obvious blocks are gone, leaving us with a bunch of small stuff (if two obvious chunks were “the dream”, a bunch of small stuff is “the nightmare”). We can still see that createPkgGraph is still the most obvious chunk, lending to the fact that we didn’t fix the fact that it’s quadratic. But, if we fix that, that’ll be a few more seconds saved! And, we can profile it again, and maybe we can look into sequenceGraph or getAllProjects, the next big chunks.

Recapping

To recap, we:

• Ran pnpm on a huge monorepo, and found it to be suspiciously slow, visibly hanging at times.
• Ran pprof-it to take a look under the hood.
• Found a couple of big candidates for optimization.
• Stared at some code.
• Got lucky, addressing both problems by simply shifting some code around.
• Made pnpm 4x faster! (For this super ridiculous test case, anyway.)

I hope this was informative. Profiling is an excellent trick to have in your toolbox. Sometimes, you’ll be unlucky and it won’t show you much. But, when you do find something, it’s worth having spent a few minutes trying it out.

In case you’re curious what else we’ve (me and the TypeScript team) have been able to find, check out these PRs and issues:

• A performance boost from avoiding the calculation of all properties of unions / intersections where all we wanted to know is if any type matches a condition.
• A performance boost by discovering that a computation was not being cached.
• A performance regression I (unwittingly) introduced in TypeScript’s string template literals when used with intersections, with two PRs (#53406 and #53413) attempting to address it.
• A performance boost in TypeScript 5.0, where I identified that we weren’t reusing our “printers” as much as we could have, saving a few percent (and even more in some projects).
• An older PR where pprof had pointed out that a lot of time during a build of a TypeScript project was being spent normalizing paths, even if the platform was UNIX-like and the paths were already using the correct slashes.
• A PR I sent back when I was working on Pylance/pyright, where 50% of GC time was spent concatenating strings.

What is “DefinitelyTyped”?

Generally speaking, there are two categories of packages on npm:

1. Packages authored in TypeScript.
2. Packages authored in JavaScript.

Whenever you install a package authored in TypeScript, you’ll also get its types.¹ This means that when you import it in your own project, you’ll get the types that the authors wrote in their code. This is the easy path; the hard work is done!

But what if the package wasn’t written in TypeScript? In this situation, it may be the case that the author hand-wrote types for their package, but most of the time, you’ll have to install types separately. It’s likely you’ve installed a package like @types/node or @types/react.

Packages published under the @types scope come from “DefinitelyTyped”, aka “DT”. DT is huge, comprising of 8,000+ packages, 6,000+ package owners, and 17,000+ unique contributors since its inception in 2012. Operating at this scale is hard, but the infrastructure is powerful enough to automate most PRs and automatically publish these packages every half hour.

How is DT laid out?

In the DT repo, there’s a directory named “types”, and that directory has all 8,000+ packages. With so many packages, you’d expect this to be one of those newfangled “monorepos” everyone’s been talking about. And it is! Well, kinda.

It turns out that even though there are over 8,000 packages in the repo, there are only about 1,200 package.json files. What gives? How does anything work?

Let’s look a file that every package does have; tsconfig.json. Here’s the tsconfig for @types/minimist:

{
    "compilerOptions": {
        "module": "commonjs",
        "lib": [
            "es6"
        ],
        "noImplicitAny": true,
        "noImplicitThis": true,
        "strictNullChecks": true,
        "strictFunctionTypes": true,
        "baseUrl": "../",
        "typeRoots": [
            "../"
        ],
        "types": [],
        "noEmit": true,
        "forceConsistentCasingInFileNames": true
    },
    "files": [
        "index.d.ts",
        "minimist-tests.ts"
    ]
}

Pretty standard stuff, but this is the critical subset:

{
    "compilerOptions": {
        "baseUrl": "../",
        "typeRoots": [
            "../"
        ],
        "types": []
    }
}

What does this do?

• baseUrl defines a path where TypeScript is allowed to perform absolute lookups. So if this package were to write import _ from "lodash", TypeScript will look for that in the types directory.
• typeRoots tells TypeScript to consider the types directory to be the @types directory that would typically be in node_modules; now, it can find @types/lodash as /types/lodash!
• types configures which @types packages are automatically included in the compilation. This can be convenient for typical projects since installing @types/node will declare all of Node’s packages and ambient types. But on DT, this is a bad idea, because we’d pull every @types package in. Setting this to the empty array stops this and allows us to manually pull things in with /// <reference types="...">.

The combined result is that DT works like a monorepo already, just without the involvement of a package manager (for the most part). If a package depends on another DT package, the publisher detects that import and automatically adds a dependency to the final package when publishing to npm.

And so, DT is a monorepo, but, it also isn’t, at least not in the way that people have come to know most monorepos in the JS world.

Of course, there are exceptions to every rule. A small fraction (~15%) of DT do have package.json files. This is because some packages depend on the types of packages not in DefinitelyTyped. This makes sense; a lot of packages are now written in TypeScript directly, and so publish their types directly, without involving DT. If a package typed on DT depends on a package that already has types, then the DT types will likely need types from that dependency as well.

What’s the problem?

It turns out that we’ve recently felt the need to change the status quo, for at least two reasons.

Firstly, since each package with a package.json needs its own external dependencies, we need to run npm install. But, we’re not a monorepo! This turns into over 30 minutes of just looping over every folder with a package.json and running npm install. Recently (as of writing), we’ve had issues with the install step randomly timing out. It’s really frustrating for the TypeScript team as we test all of DefinitelyTyped on most type checking changes, just to make sure we don’t break anyone (or, only break things in desirable ways).

Secondly, you may remember that the tsconfig.json from earlier set "module": "commonjs". This is the only valid configuration on DT and it has worked for a very long time. But as more and more packages start using features like ESM and export maps, DT needs to be able to support those features. And it does! Mostly. The "module": "commonjs" lie can be worked around for the most part, but DT should really be set to "moduleResolution": "node16" and then actually test that the packages and their dependencies and dependants actually work in that more modern mode.

A solution to both of these problems is to turn DefinitelyTyped into a monorepo more like what other major projects are doing, meaning:

• Add a package.json to every DT package.
• Explicitly declare all dependencies, even those within the repo.
• Let a package manager or monorepo tool link the projects in node_modules.
• Install everything at once.
• Drop baseUrl and typeRoots out of every tsconfig.json.

This (theoretically) gets us a much faster install time, as well as getting us a final state on disk that matches what downstream users see, enabling packages to start making use of "moduleResolution": "node16".

What next?

This is a cool idea in theory, but to make it real, we have to make some choices. Specifically, the tooling. There are some unique restrictions which make this choice complicated:

• The tool has to be handle the 8,000+ DT packages and their external dependencies.
• The tool shouldn’t hoist anything, unless it’s safe to do so. We don’t want to accidentally resolve anything.
• The tool must be able to handle multiple versions of packages in types (e.g. @types/react in types/react, @types/react@v17 in types/react/v17, and so on).
• The tool should be fast. Right now, if you work on one package, you may not even need to install anything. If you do install a package, you’re only going to pay for the cost of installing that one DT package’s deps. If we have to get the whole monorepo, that experience hopefully shouldn’t suffer.
• The tool shouldn’t try and do anything else. We just want package linking, not a build system. There’s nothing to build!

This set of requirements really narrows it down; at the time of writing, the only package manager which meets these requirements is pnpm. The other choices either ban packages of duplicated names, are generally not configurable enough, or take too long to install (though no option is likely slower than the 30 minute CI install). I’m not super surprised; pnpm is the default package manager of the rushstack tooling and there are some pretty ridiculously sized monorepos using it.

Even still, pnpm’s great performance still felt a little slow. I noticed that on install it’d hang and then start printing text, implying some performance problem. Not shocking; the number of packages it finally resolves to is over 9,000, and I’d think any tool would chug with that much work to do.

But, there’s good news! By profiling pnpm install, I discovered that that the performance holes are mostly just cases of “accidentally quadratic” code, and therefore can be addressed.

And that’s the actual thing I wanted to write about before I got carried away. For details on that, check out the next post in this series.

👋 package manager maintainers

There’s no doubt in my mind that this post will eventually make its way to the maintainers of the other package managers and monorepo tools. Understand, I really truly do not mean anything negative in the above. I use all of your tools and they’re all great! My focus on pnpm above is due to the fact that I’m able to make immediate progress with it and that it also lets me demo some cool profiling techniques I’ve been meaning to share for a while. I have no idea how DT will actually end up, we’re just hurting now and I’m finding this fun to play around with.