Expand auto-import to all package.json dependencies by andrewbranch · Pull Request #38923 · microsoft/TypeScript (original) (raw)

This PR creates an auxiliary program in the language service layer that contains node_modules packages that are specified as a direct dependency (including devDependencies peerDependencies) in a package.json file visible to the project’s root directory and not already present in the main program. This program is then supplied to the auto-imports machinery as a provider of additional modules that can be used for auto-import. Effectively, this means that packages that ship their own types should be available for auto-import immediately after npm installing them, lifting a long-despised limitation of “you have to write out the import manually once in your project before auto-imports will work, unless it’s in @types, because @types is special.”

Performance

I compared a few operations against master on the output of @angular/cli (with routing). (I chose this as a test because it’s a realistic scenario where a user’s package.json will contain more dependencies with self-shipped types than are actually used in the program. For example, @angular/forms is a listed dependency but isn’t used in the boilerplate.) Measurements are an average of three trials run in VS Code 1.46.0-insider (b1ef2bf) with extensions disabled.

* This triggers completions, including import suggestions for PatternValidator from @angular/forms in the PR trials, a suggestion that is not included in the master trials.

** This triggers completions again without disrupting the auto-import cache populated in the previous action.

*** This changes the structure of the primary program, which triggers an update of the auto-import provider program. The time it takes to do so is included in the subsequent getApplicableRefactors measurement.

Observations:

• Editing operations that don’t change the structure of the program are unaffected (looking at updateGraph time)
• When generating completions, there’s an added cost associated with looking through more modules and generating auto-import suggestions for them. The penalty isn’t huge, though, and seems to be worth it for the added utility.
• When something happens that triggers an update of the auto-import provider, there’s an associated cost that scales with the number of typed dependencies in package.json files that are not already in the main program (i.e., the size of the auto-import provider program). Because the Angular CLI generates boilerplate that has more unused typed dependencies than most real codebases (normally when you install a dependency, you import it shortly thereafter), I suspect that it will be rare for users to see much more delay than the 18 ms measured here.

Limitations

• Only package.jsons found in the project’s currentDirectory and upward are considered. So in an unusual project structure like

tsconfig.json  
project/  
  node_modules/  
  package.json  
  index.ts  

the project/package.json file won’t be found.

• This doesn’t currently resolve to JavaScript files—that is, to get auto imports, the package has to have .d.ts files. Processing plain JS for auto-imports could be explored if demand were high, but when I tried it, most of the added packages were CLI tools and plugins, not things people want to import from, so it was a performance hit for little (or even negative) utility.

Memory considerations

The auxiliary program is configured to be as light as possible, but in a project references editing scenario, it’s possible to have many projects open at once. Each project has its own language service, so each open project will typically have one of these auxiliary programs. Some possibilities for mitigating high memory usage in these scenarios: (updated: decided to exclude devDependencies for UX reasons)

• The auxiliary program won’t be created if there are no packages listed that aren’t already in the main program, but an analysis of around 500 popular open source TypeScript repos shows that this rarely happens, largely due to devDependencies that ship their own types but are never used in the program (prominent example: typescript). Excluding devDependencies significantly increases the chances that creation an auxiliary program can be skipped, at the cost of those devDependencies being unavailable for auto-import (which can be useful when writing build scripts and tests).
• The auxiliary program isn’t created until there’s an open file that belongs¹ to the project. This prevents us from creating these programs as we do a cross-project find-all-references.
• If we detect that memory usage is a problem, we could dispose or stop creating these programs. I think @sheetalkamat has looked into this in the past.
• We could either hard-code into TypeScript or indicate in third-party projects that a package should opt out of being eagerly auto-importable. In one real-world project I tested, the contents of the auxiliary programs were limited to typescript, typescript-eslint, ts-jest, jest, and prettier, none of which are commonly imported.

However, at this point I’m not convinced that any of these mitigations are necessary. The size of these programs is usually in the ballpark of a browser tab (around 50 MB from one early test). But we have avenues to explore if needed.

Fixes #37812
Fixes #36042

¹ “file that belongs to the project” here means “file whose default project is the project in question,” in contrast to “file that is contained by the project”