Monomorphize allocators for tsserver/public API, just like core compiler by jakebailey · Pull Request #58045 · microsoft/TypeScript (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation79 Commits5 Checks25 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

jakebailey

In #51682 and #51880, @rbuckton monomorphized our Nodes and Symbols for the compiler's allocators.

This PR naively copies the property order from the compiler over to the classes we define in services, netting similar performance wins for similar memory costs. This should significantly improve performance of tsserver (every editor) and public API users who do anything more than parse (ts-eslint, ts-loader, ts-jest, rush, nx, the list goes on...).

Here are the latest benchmarks when forcing tsc to use these allocators, which is a good proxy for how this will affect downstream users: #58045 (comment)

Note that this is not a performance improvement for tsc itself; that benefit already happened in TS 5.0!

image

The tsserver benchmarks remain, and should lead to significant improvements in editor performance:

image

A new benchmark tests "tsc via public API" explicitly:

image

RyanCavanaugh, JoshuaKGoldberg, fabiospampinato, aaronccasanova, robpalme, SlurpTheo, Kobzol, and anuraghazra reacted with thumbs up emoji RyanCavanaugh, JoshuaKGoldberg, fabiospampinato, and anuraghazra reacted with hooray emoji DanielRosenwasser, danvk, fabiospampinato, and anuraghazra reacted with heart emoji RyanCavanaugh, JoshuaKGoldberg, fabiospampinato, and anuraghazra reacted with rocket emoji

@jakebailey

@typescript-bot

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

jakebailey

@jakebailey

After the results come back, I'll try and rebase this PR such that it targets a branch that has the tsc change in place. The benchmark system is smart enough to compare a PR against its target branch so that will provide better results than comparing two negative results to see which one is less negative.

(I am liking what I'm seeing in the logs so far, though.)

@typescript-bot

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,662k (± 0.01%) 301,824k (± 0.01%) +6,162k (+ 2.08%) 301,774k 301,845k p=0.005 n=6
Parse Time 3.19s (± 0.65%) 3.24s (± 0.80%) +0.04s (+ 1.36%) 3.21s 3.28s p=0.015 n=6
Bind Time 1.00s (± 0.82%) 0.99s (± 0.52%) ~ 0.99s 1.00s p=0.523 n=6
Check Time 9.76s (± 0.26%) 9.81s (± 0.38%) +0.05s (+ 0.55%) 9.78s 9.88s p=0.028 n=6
Emit Time 8.38s (± 0.22%) 8.44s (± 0.30%) +0.06s (+ 0.76%) 8.41s 8.47s p=0.005 n=6
Total Time 22.33s (± 0.20%) 22.49s (± 0.24%) +0.15s (+ 0.67%) 22.41s 22.56s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 193,378k (± 0.94%) 199,743k (± 0.94%) +6,365k (+ 3.29%) 197,382k 202,031k p=0.005 n=6
Parse Time 1.65s (± 0.92%) 1.69s (± 1.24%) +0.04s (+ 2.74%) 1.67s 1.72s p=0.005 n=6
Bind Time 0.88s (± 0.86%) 0.89s (± 0.58%) ~ 0.88s 0.89s p=0.069 n=6
Check Time 11.25s (± 0.22%) 11.30s (± 0.65%) ~ 11.21s 11.40s p=0.422 n=6
Emit Time 3.16s (± 0.31%) 3.19s (± 5.20%) ~ 3.02s 3.36s p=1.000 n=6
Total Time 16.93s (± 0.22%) 17.06s (± 1.06%) ~ 16.81s 17.26s p=0.297 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,426k (± 0.01%) 353,378k (± 0.01%) +5,952k (+ 1.71%) 353,343k 353,466k p=0.005 n=6
Parse Time 3.70s (± 1.33%) 3.60s (± 1.05%) -0.10s (- 2.66%) 3.56s 3.66s p=0.016 n=6
Bind Time 1.37s (± 0.38%) 1.59s (± 3.06%) 🔻+0.22s (+16.02%) 1.50s 1.64s p=0.004 n=6
Check Time 10.20s (± 0.18%) 10.18s (± 0.38%) ~ 10.13s 10.24s p=0.228 n=6
Emit Time 6.03s (± 0.29%) 6.05s (± 0.59%) ~ 6.02s 6.12s p=0.145 n=6
Total Time 21.30s (± 0.18%) 21.43s (± 0.46%) +0.13s (+ 0.59%) 21.30s 21.60s p=0.020 n=6
TFS - node (v18.15.0, x64)
Memory used 302,830k (± 0.01%) 308,523k (± 0.01%) +5,693k (+ 1.88%) 308,500k 308,553k p=0.005 n=6
Parse Time 2.42s (± 0.61%) 2.44s (± 1.03%) ~ 2.42s 2.48s p=0.075 n=6
Bind Time 1.20s (± 0.43%) 1.16s (±10.27%) ~ 1.00s 1.25s p=0.366 n=6
Check Time 7.55s (± 0.61%) 7.62s (± 1.52%) ~ 7.52s 7.78s p=0.469 n=6
Emit Time 4.27s (± 0.52%) 4.31s (± 0.64%) ~ 4.28s 4.35s p=0.052 n=6
Total Time 15.44s (± 0.31%) 15.54s (± 0.22%) +0.10s (+ 0.62%) 15.50s 15.59s p=0.016 n=6
material-ui - node (v18.15.0, x64)
Memory used 510,164k (± 0.01%) 517,146k (± 0.00%) +6,982k (+ 1.37%) 517,117k 517,184k p=0.005 n=6
Parse Time 3.93s (± 0.35%) 3.98s (± 1.13%) +0.06s (+ 1.40%) 3.95s 4.06s p=0.009 n=6
Bind Time 1.45s (± 0.83%) 1.50s (± 1.31%) +0.05s (+ 3.44%) 1.47s 1.52s p=0.006 n=6
Check Time 25.41s (± 0.42%) 25.36s (± 0.42%) ~ 25.23s 25.51s p=0.470 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 30.79s (± 0.35%) 30.84s (± 0.40%) ~ 30.67s 31.03s p=0.575 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,744,999k (± 0.00%) 1,750,707k (± 0.00%) +5,708k (+ 0.33%) 1,750,659k 1,750,781k p=0.005 n=6
Parse Time 7.80s (± 0.21%) 8.02s (± 2.40%) +0.22s (+ 2.84%) 7.91s 8.41s p=0.004 n=6
Bind Time 2.82s (± 0.30%) 2.29s (± 1.42%) 🟩-0.53s (-18.88%) 2.27s 2.35s p=0.004 n=6
Check Time 67.07s (± 0.44%) 68.89s (± 1.49%) +1.82s (+ 2.72%) 67.54s 69.74s p=0.005 n=6
Emit Time 0.16s (± 2.52%) 0.16s (± 2.58%) ~ 0.15s 0.16s p=0.218 n=6
Total Time 77.86s (± 0.38%) 79.36s (± 1.11%) +1.50s (+ 1.93%) 78.05s 80.15s p=0.008 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,399,823k (± 0.02%) 2,489,901k (± 0.03%) +90,078k (+ 3.75%) 2,489,254k 2,490,841k p=0.005 n=6
Parse Time 6.02s (± 0.70%) 6.29s (± 0.90%) 🔻+0.27s (+ 4.43%) 6.20s 6.35s p=0.005 n=6
Bind Time 2.26s (± 1.10%) 2.04s (± 1.50%) 🟩-0.22s (- 9.72%) 1.99s 2.07s p=0.005 n=6
Check Time 39.95s (± 0.62%) 40.49s (± 0.15%) +0.55s (+ 1.37%) 40.43s 40.60s p=0.005 n=6
Emit Time 3.17s (± 1.04%) 3.18s (± 1.41%) ~ 3.12s 3.23s p=0.810 n=6
Total Time 51.43s (± 0.39%) 52.01s (± 0.23%) +0.58s (+ 1.12%) 51.85s 52.12s p=0.005 n=6
self-compiler - node (v18.15.0, x64)
Memory used 416,131k (± 0.01%) 421,766k (± 0.02%) +5,635k (+ 1.35%) 421,693k 421,940k p=0.005 n=6
Parse Time 2.76s (± 0.37%) 2.74s (± 2.70%) ~ 2.65s 2.82s p=0.418 n=6
Bind Time 1.08s (± 0.38%) 1.16s (± 4.12%) 🔻+0.08s (+ 7.73%) 1.09s 1.22s p=0.004 n=6
Check Time 15.42s (± 0.37%) 15.47s (± 0.34%) ~ 15.41s 15.56s p=0.109 n=6
Emit Time 1.12s (± 1.35%) 1.12s (± 2.24%) ~ 1.09s 1.15s p=0.935 n=6
Total Time 20.38s (± 0.33%) 20.49s (± 0.28%) +0.12s (+ 0.58%) 20.43s 20.60s p=0.019 n=6
vscode - node (v18.15.0, x64)
Memory used 2,899,596k (± 0.00%) 2,901,558k (± 0.00%) +1,962k (+ 0.07%) 2,901,446k 2,901,653k p=0.005 n=6
Parse Time 10.84s (± 0.43%) 11.08s (± 0.51%) +0.24s (+ 2.20%) 11.03s 11.17s p=0.005 n=6
Bind Time 3.48s (± 0.82%) 3.46s (± 0.35%) ~ 3.45s 3.48s p=0.411 n=6
Check Time 61.97s (± 0.74%) 62.10s (± 0.34%) ~ 61.82s 62.46s p=0.173 n=6
Emit Time 16.46s (± 0.52%) 16.58s (± 0.21%) ~ 16.54s 16.63s p=0.054 n=6
Total Time 92.75s (± 0.51%) 93.23s (± 0.22%) ~ 92.90s 93.51s p=0.066 n=6
webpack - node (v18.15.0, x64)
Memory used 408,862k (± 0.01%) 415,521k (± 0.01%) +6,659k (+ 1.63%) 415,441k 415,603k p=0.005 n=6
Parse Time 3.24s (± 0.30%) 3.33s (± 0.75%) +0.10s (+ 2.99%) 3.31s 3.38s p=0.004 n=6
Bind Time 1.41s (± 0.73%) 1.38s (± 0.59%) -0.02s (- 1.66%) 1.37s 1.39s p=0.008 n=6
Check Time 14.31s (± 0.34%) 14.44s (± 0.13%) +0.13s (+ 0.87%) 14.42s 14.46s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 18.96s (± 0.20%) 19.16s (± 0.11%) +0.20s (+ 1.05%) 19.12s 19.18s p=0.005 n=6
xstate - node (v18.15.0, x64)
Memory used 513,453k (± 0.02%) 531,453k (± 0.01%) +18,000k (+ 3.51%) 531,373k 531,524k p=0.005 n=6
Parse Time 3.93s (± 0.47%) 3.98s (± 0.37%) +0.05s (+ 1.14%) 3.95s 3.99s p=0.012 n=6
Bind Time 1.88s (± 0.65%) 1.67s (± 1.22%) 🟩-0.21s (-11.02%) 1.64s 1.69s p=0.005 n=6
Check Time 3.42s (± 1.02%) 3.60s (± 0.82%) 🔻+0.19s (+ 5.46%) 3.55s 3.64s p=0.005 n=6
Emit Time 0.09s (± 5.95%) 0.08s (± 4.99%) ~ 0.08s 0.09s p=0.112 n=6
Total Time 9.32s (± 0.45%) 9.33s (± 0.60%) ~ 9.23s 9.39s p=0.470 n=6

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,562ms (± 0.27%) 3,524ms (± 0.44%) -38ms (- 1.07%) 3,504ms 3,547ms p=0.005 n=6
Req 2 - geterr 8,322ms (± 0.64%) 7,603ms (± 0.46%) 🟩-719ms (- 8.64%) 7,553ms 7,647ms p=0.005 n=6
Req 3 - references 475ms (± 0.34%) 470ms (± 1.05%) -5ms (- 1.09%) 463ms 475ms p=0.036 n=6
Req 4 - navto 407ms (± 0.18%) 410ms (± 2.59%) ~ 398ms 427ms p=0.373 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 129ms (± 1.44%) 112ms (± 1.31%) 🟩-17ms (-13.27%) 111ms 114ms p=0.004 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,223ms (±11.44%) 3,301ms (±11.73%) ~ 2,935ms 3,675ms p=0.471 n=6
Req 2 - geterr 6,135ms (± 1.48%) 5,302ms (±10.57%) 🟩-833ms (-13.58%) 4,570ms 5,729ms p=0.005 n=6
Req 3 - references 472ms (± 6.82%) 441ms (± 9.07%) ~ 379ms 475ms p=0.126 n=6
Req 4 - navto 430ms (± 7.48%) 390ms (±10.69%) 🟩-41ms (- 9.49%) 337ms 434ms p=0.030 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 108ms (±10.76%) 109ms (± 1.00%) ~ 107ms 110ms p=1.000 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,634ms (± 0.20%) 2,596ms (± 0.63%) -39ms (- 1.47%) 2,577ms 2,614ms p=0.005 n=6
Req 2 - geterr 1,739ms (± 2.95%) 1,539ms (± 0.83%) 🟩-200ms (-11.49%) 1,521ms 1,551ms p=0.005 n=6
Req 3 - references 117ms (± 9.69%) 114ms (± 5.65%) ~ 101ms 118ms p=0.297 n=6
Req 4 - navto 369ms (± 1.12%) 365ms (± 1.47%) ~ 357ms 374ms p=0.052 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 307ms (± 2.60%) 291ms (± 0.49%) 🟩-16ms (- 5.06%) 289ms 293ms p=0.005 n=6

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 182.54ms (± 0.18%) 277.41ms (± 0.31%) 🔻+94.87ms (+51.97%) 269.96ms 286.39ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 226.49ms (± 0.17%) 227.25ms (± 0.20%) +0.76ms (+ 0.34%) 225.57ms 233.91ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 221.66ms (± 0.15%) 222.40ms (± 0.16%) +0.74ms (+ 0.33%) 220.79ms 227.46ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 330.05ms (± 0.32%) 331.23ms (± 0.30%) +1.18ms (+ 0.36%) 322.97ms 340.80ms p=0.000 n=600

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey jakebailey changed the base branch from main to jabaile/tsc-services

April 3, 2024 00:01

@jakebailey

@typescript-bot

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@jakebailey

@typescript-bot

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 288,737k (± 0.02%) 301,845k (± 0.01%) 🔻+13,108k (+ 4.54%) 301,820k 301,897k p=0.005 n=6
Parse Time 3.25s (± 0.42%) 3.24s (± 0.60%) ~ 3.22s 3.27s p=0.742 n=6
Bind Time 1.01s (± 1.36%) 0.99s (± 0.41%) -0.02s (- 2.30%) 0.99s 1.00s p=0.006 n=6
Check Time 10.64s (± 0.38%) 9.81s (± 0.18%) 🟩-0.83s (- 7.77%) 9.79s 9.84s p=0.005 n=6
Emit Time 9.44s (± 0.26%) 8.43s (± 0.32%) 🟩-1.02s (-10.75%) 8.39s 8.45s p=0.005 n=6
Total Time 24.34s (± 0.27%) 22.48s (± 0.21%) 🟩-1.87s (- 7.67%) 22.43s 22.55s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 195,077k (± 0.95%) 199,158k (± 0.78%) +4,081k (+ 2.09%) 197,450k 201,168k p=0.008 n=6
Parse Time 2.07s (± 1.27%) 2.07s (± 1.13%) ~ 2.03s 2.09s p=0.466 n=6
Bind Time 1.10s (± 0.50%) 1.09s (± 1.38%) ~ 1.07s 1.11s p=0.054 n=6
Check Time 15.14s (± 0.45%) 14.02s (± 0.77%) 🟩-1.12s (- 7.41%) 13.88s 14.20s p=0.005 n=6
Emit Time 4.29s (± 1.04%) 3.94s (± 5.00%) 🟩-0.35s (- 8.16%) 3.75s 4.13s p=0.005 n=6
Total Time 22.61s (± 0.50%) 21.12s (± 0.92%) 🟩-1.49s (- 6.58%) 20.94s 21.42s p=0.005 n=6
Monaco - node (v18.15.0, x64)
Memory used 332,718k (± 0.01%) 353,354k (± 0.01%) 🔻+20,637k (+ 6.20%) 353,322k 353,417k p=0.005 n=6
Parse Time 3.57s (± 0.60%) 3.64s (± 2.99%) ~ 3.54s 3.85s p=0.107 n=6
Bind Time 1.72s (± 0.44%) 1.55s (± 6.46%) 🟩-0.17s (- 9.87%) 1.36s 1.61s p=0.004 n=6
Check Time 11.44s (± 0.27%) 10.18s (± 0.27%) 🟩-1.26s (-11.04%) 10.14s 10.21s p=0.005 n=6
Emit Time 6.68s (± 0.26%) 6.06s (± 0.45%) 🟩-0.62s (- 9.28%) 6.03s 6.11s p=0.005 n=6
Total Time 23.42s (± 0.22%) 21.43s (± 0.25%) 🟩-1.99s (- 8.48%) 21.36s 21.48s p=0.005 n=6
TFS - node (v18.15.0, x64)
Memory used 295,350k (± 0.08%) 308,515k (± 0.01%) 🔻+13,166k (+ 4.46%) 308,473k 308,548k p=0.005 n=6
Parse Time 2.00s (± 0.49%) 2.04s (± 0.88%) +0.04s (+ 1.92%) 2.02s 2.07s p=0.005 n=6
Bind Time 1.05s (± 0.72%) 1.03s (± 0.40%) -0.02s (- 1.90%) 1.03s 1.04s p=0.004 n=6
Check Time 7.25s (± 0.23%) 6.40s (± 0.43%) 🟩-0.85s (-11.74%) 6.37s 6.43s p=0.005 n=6
Emit Time 4.11s (± 0.45%) 3.60s (± 0.41%) 🟩-0.51s (-12.38%) 3.58s 3.61s p=0.005 n=6
Total Time 14.41s (± 0.25%) 13.07s (± 0.32%) 🟩-1.34s (- 9.30%) 13.01s 13.11s p=0.005 n=6
material-ui - node (v18.15.0, x64)
Memory used 488,586k (± 0.01%) 517,173k (± 0.01%) 🔻+28,587k (+ 5.85%) 517,120k 517,219k p=0.005 n=6
Parse Time 3.21s (± 0.75%) 3.25s (± 0.32%) +0.04s (+ 1.09%) 3.23s 3.26s p=0.048 n=6
Bind Time 1.27s (± 7.13%) 1.23s (± 1.26%) ~ 1.22s 1.26s p=0.063 n=6
Check Time 22.16s (± 0.41%) 20.62s (± 0.16%) 🟩-1.54s (- 6.95%) 20.58s 20.65s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 26.65s (± 0.32%) 25.09s (± 0.07%) 🟩-1.55s (- 5.82%) 25.07s 25.12s p=0.005 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,654,339k (± 0.00%) 1,750,659k (± 0.00%) 🔻+96,320k (+ 5.82%) 1,750,577k 1,750,724k p=0.005 n=6
Parse Time 9.59s (± 0.43%) 9.89s (± 1.77%) +0.30s (+ 3.18%) 9.76s 10.24s p=0.005 n=6
Bind Time 2.94s (± 0.88%) 2.79s (± 0.95%) 🟩-0.15s (- 5.04%) 2.76s 2.83s p=0.005 n=6
Check Time 91.27s (± 1.63%) 83.26s (± 1.14%) 🟩-8.01s (- 8.78%) 82.37s 84.57s p=0.005 n=6
Emit Time 0.19s (± 2.67%) 0.19s (± 2.81%) ~ 0.19s 0.20s p=0.640 n=6
Total Time 103.99s (± 1.42%) 96.14s (± 1.08%) 🟩-7.86s (- 7.55%) 95.21s 97.59s p=0.005 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,366,559k (± 0.12%) 2,490,452k (± 0.05%) 🔻+123,893k (+ 5.24%) 2,488,554k 2,492,257k p=0.005 n=6
Parse Time 7.67s (± 0.78%) 7.80s (± 1.04%) +0.13s (+ 1.67%) 7.67s 7.89s p=0.031 n=6
Bind Time 2.63s (± 0.51%) 2.52s (± 0.70%) 🟩-0.11s (- 4.31%) 2.50s 2.54s p=0.005 n=6
Check Time 53.33s (± 0.68%) 50.04s (± 0.17%) 🟩-3.29s (- 6.17%) 49.95s 50.20s p=0.005 n=6
Emit Time 4.42s (± 8.22%) 3.96s (± 0.65%) 🟩-0.46s (-10.45%) 3.92s 3.99s p=0.005 n=6
Total Time 68.05s (± 0.56%) 64.34s (± 0.23%) 🟩-3.71s (- 5.45%) 64.18s 64.55s p=0.005 n=6
self-compiler - node (v18.15.0, x64)
Memory used 406,697k (± 0.01%) 421,757k (± 0.01%) +15,060k (+ 3.70%) 421,728k 421,790k p=0.005 n=6
Parse Time 3.22s (± 0.61%) 3.24s (± 0.53%) ~ 3.22s 3.26s p=0.139 n=6
Bind Time 1.58s (± 0.67%) 1.43s (± 1.51%) 🟩-0.14s (- 8.89%) 1.41s 1.46s p=0.005 n=6
Check Time 19.50s (± 0.38%) 18.20s (± 0.26%) 🟩-1.30s (- 6.68%) 18.13s 18.27s p=0.005 n=6
Emit Time 1.37s (± 1.56%) 1.35s (± 1.66%) ~ 1.32s 1.38s p=0.332 n=6
Total Time 25.66s (± 0.34%) 24.22s (± 0.26%) 🟩-1.44s (- 5.60%) 24.14s 24.32s p=0.005 n=6
vscode - node (v18.15.0, x64)
Memory used 2,759,368k (± 0.00%) 2,901,769k (± 0.00%) 🔻+142,401k (+ 5.16%) 2,901,690k 2,901,833k p=0.005 n=6
Parse Time 11.03s (± 0.39%) 11.08s (± 0.49%) ~ 11.00s 11.14s p=0.125 n=6
Bind Time 3.62s (± 0.39%) 3.47s (± 0.50%) 🟩-0.15s (- 4.05%) 3.46s 3.50s p=0.005 n=6
Check Time 68.59s (± 0.52%) 62.22s (± 0.27%) 🟩-6.38s (- 9.30%) 61.92s 62.37s p=0.005 n=6
Emit Time 18.21s (± 0.35%) 16.59s (± 0.36%) 🟩-1.62s (- 8.87%) 16.54s 16.70s p=0.005 n=6
Total Time 101.45s (± 0.38%) 93.36s (± 0.17%) 🟩-8.09s (- 7.97%) 93.08s 93.50s p=0.005 n=6
webpack - node (v18.15.0, x64)
Memory used 396,884k (± 0.02%) 415,541k (± 0.01%) 🔻+18,657k (+ 4.70%) 415,460k 415,612k p=0.005 n=6
Parse Time 3.19s (± 1.63%) 3.31s (± 1.23%) +0.11s (+ 3.60%) 3.26s 3.38s p=0.008 n=6
Bind Time 1.47s (± 0.35%) 1.38s (± 0.40%) 🟩-0.08s (- 5.57%) 1.38s 1.39s p=0.004 n=6
Check Time 15.29s (± 0.24%) 14.45s (± 0.32%) 🟩-0.85s (- 5.53%) 14.36s 14.48s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 19.96s (± 0.19%) 19.14s (± 0.42%) 🟩-0.81s (- 4.08%) 19.00s 19.24s p=0.005 n=6
xstate - node (v18.15.0, x64)
Memory used 494,548k (± 0.03%) 531,647k (± 0.02%) 🔻+37,099k (+ 7.50%) 531,487k 531,805k p=0.005 n=6
Parse Time 4.93s (± 0.25%) 4.96s (± 0.52%) +0.03s (+ 0.57%) 4.94s 5.01s p=0.029 n=6
Bind Time 2.21s (± 1.20%) 2.08s (± 0.83%) 🟩-0.14s (- 6.10%) 2.05s 2.10s p=0.005 n=6
Check Time 4.68s (± 1.04%) 4.48s (± 0.85%) 🟩-0.20s (- 4.20%) 4.44s 4.53s p=0.005 n=6
Emit Time 0.11s (± 0.00%) 0.10s (± 0.00%) 🟩-0.01s (- 9.09%) 0.10s 0.10s p=0.001 n=6
Total Time 11.94s (± 0.54%) 11.63s (± 0.50%) -0.31s (- 2.60%) 11.54s 11.68s p=0.005 n=6

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,369ms (± 0.49%) 2,369ms (± 0.88%) ~ 2,335ms 2,395ms p=0.873 n=6
Req 2 - geterr 5,629ms (± 0.43%) 5,131ms (± 0.55%) 🟩-498ms (- 8.85%) 5,078ms 5,161ms p=0.005 n=6
Req 3 - references 318ms (± 0.47%) 311ms (± 1.42%) -7ms (- 2.20%) 308ms 317ms p=0.011 n=6
Req 4 - navto 271ms (± 0.60%) 269ms (± 3.64%) -2ms (- 0.86%) 265ms 289ms p=0.047 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 89ms (± 4.97%) 77ms (± 0.72%) 🟩-13ms (-14.04%) 76ms 77ms p=0.004 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,721ms (± 0.68%) 3,672ms (± 0.93%) ~ 3,643ms 3,734ms p=0.054 n=6
Req 2 - geterr 6,110ms (± 0.27%) 5,647ms (± 0.51%) 🟩-463ms (- 7.57%) 5,598ms 5,682ms p=0.005 n=6
Req 3 - references 494ms (± 0.30%) 465ms (± 0.46%) 🟩-29ms (- 5.77%) 462ms 468ms p=0.005 n=6
Req 4 - navto 446ms (± 0.63%) 415ms (± 1.76%) 🟩-31ms (- 6.85%) 402ms 422ms p=0.004 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 108ms (± 0.95%) 109ms (± 0.95%) ~ 107ms 110ms p=0.652 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,351ms (±10.16%) 3,096ms (± 0.58%) 🟩-255ms (- 7.61%) 3,076ms 3,123ms p=0.013 n=6
Req 2 - geterr 2,380ms (± 5.52%) 2,127ms (±10.43%) 🟩-254ms (-10.66%) 1,832ms 2,280ms p=0.031 n=6
Req 3 - references 149ms (± 1.84%) 139ms (± 1.56%) 🟩-10ms (- 6.62%) 136ms 142ms p=0.005 n=6
Req 4 - navto 552ms (± 0.34%) 537ms (± 1.30%) -15ms (- 2.75%) 532ms 551ms p=0.010 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 455ms (± 1.87%) 443ms (± 0.84%) -12ms (- 2.67%) 438ms 449ms p=0.015 n=6

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 228.74ms (± 0.15%) 228.80ms (± 0.14%) +0.06ms (+ 0.03%) 227.49ms 233.92ms p=0.014 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 227.75ms (± 0.15%) 227.71ms (± 0.21%) -0.04ms (- 0.02%) 226.21ms 236.35ms p=0.013 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 223.62ms (± 0.16%) 223.63ms (± 0.17%) ~ 221.93ms 227.33ms p=0.993 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 223.46ms (± 0.15%) 223.55ms (± 0.16%) +0.09ms (+ 0.04%) 221.95ms 225.83ms p=0.021 n=600

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey

@jakebailey

This comment was marked as outdated.

@typescript-bot

This comment was marked as outdated.

@typescript-bot

This comment was marked as outdated.

@jakebailey

This comment was marked as outdated.

@jakebailey

This comment was marked as outdated.

@jakebailey

FWIW comparing #51682 (comment), this increase memory is not unexpected. Maybe it's fine; this benchmark is already extra strange because I'm emitting stuff, which won't happen in tsserver... normally anyway.

I feel like if #51682 (comment) was "good", then this PR is good.

@jakebailey

@typescript-bot

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 288,752k (± 0.01%) 305,167k (± 0.01%) 🔻+16,415k (+ 5.68%) 305,144k 305,200k p=0.005 n=6
Parse Time 4.01s (± 0.49%) 4.03s (± 0.41%) ~ 4.00s 4.05s p=0.406 n=6
Bind Time 1.27s (± 0.77%) 1.25s (± 0.98%) -0.02s (- 1.84%) 1.23s 1.26s p=0.008 n=6
Check Time 13.19s (± 0.41%) 12.15s (± 0.67%) 🟩-1.04s (- 7.92%) 12.05s 12.24s p=0.005 n=6
Emit Time 11.76s (± 0.35%) 10.54s (± 0.53%) 🟩-1.23s (-10.42%) 10.48s 10.62s p=0.005 n=6
Total Time 30.24s (± 0.26%) 27.95s (± 0.17%) 🟩-2.29s (- 7.56%) 27.87s 28.00s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 195,094k (± 0.95%) 201,384k (± 0.97%) +6,290k (+ 3.22%) 198,954k 203,597k p=0.005 n=6
Parse Time 2.04s (± 1.15%) 2.09s (± 0.56%) +0.05s (+ 2.28%) 2.07s 2.10s p=0.011 n=6
Bind Time 1.11s (± 0.73%) 1.08s (± 1.23%) -0.03s (- 2.84%) 1.06s 1.10s p=0.006 n=6
Check Time 15.17s (± 0.66%) 13.97s (± 0.36%) 🟩-1.19s (- 7.88%) 13.93s 14.07s p=0.005 n=6
Emit Time 4.33s (± 0.92%) 3.83s (± 3.45%) 🟩-0.49s (-11.40%) 3.74s 4.10s p=0.005 n=6
Total Time 22.65s (± 0.44%) 20.97s (± 0.57%) 🟩-1.67s (- 7.40%) 20.89s 21.21s p=0.005 n=6
Monaco - node (v18.15.0, x64)
Memory used 332,724k (± 0.01%) 357,725k (± 0.01%) 🔻+25,001k (+ 7.51%) 357,705k 357,749k p=0.005 n=6
Parse Time 3.53s (± 1.17%) 3.70s (± 2.86%) 🔻+0.17s (+ 4.91%) 3.52s 3.81s p=0.020 n=6
Bind Time 1.71s (± 0.71%) 1.42s (± 6.28%) 🟩-0.30s (-17.41%) 1.33s 1.56s p=0.005 n=6
Check Time 11.41s (± 0.35%) 10.21s (± 0.52%) 🟩-1.20s (-10.51%) 10.14s 10.29s p=0.005 n=6
Emit Time 6.69s (± 0.31%) 6.06s (± 0.55%) 🟩-0.63s (- 9.44%) 6.00s 6.10s p=0.005 n=6
Total Time 23.34s (± 0.20%) 21.38s (± 0.31%) 🟩-1.96s (- 8.39%) 21.29s 21.46s p=0.005 n=6
TFS - node (v18.15.0, x64)
Memory used 295,145k (± 0.11%) 311,915k (± 0.00%) 🔻+16,769k (+ 5.68%) 311,903k 311,927k p=0.005 n=6
Parse Time 2.01s (± 0.80%) 2.04s (± 0.67%) +0.02s (+ 1.24%) 2.01s 2.05s p=0.031 n=6
Bind Time 1.06s (± 0.71%) 1.03s (± 1.19%) 🟩-0.04s (- 3.45%) 1.00s 1.03s p=0.003 n=6
Check Time 7.25s (± 0.27%) 6.39s (± 0.38%) 🟩-0.85s (-11.75%) 6.37s 6.42s p=0.005 n=6
Emit Time 4.10s (± 0.51%) 3.60s (± 0.44%) 🟩-0.50s (-12.24%) 3.58s 3.62s p=0.005 n=6
Total Time 14.42s (± 0.21%) 13.06s (± 0.34%) 🟩-1.36s (- 9.44%) 13.01s 13.12s p=0.005 n=6
material-ui - node (v18.15.0, x64)
Memory used 488,567k (± 0.00%) 519,876k (± 0.01%) 🔻+31,309k (+ 6.41%) 519,811k 519,920k p=0.005 n=6
Parse Time 3.20s (± 0.80%) 3.28s (± 1.45%) +0.08s (+ 2.50%) 3.22s 3.36s p=0.010 n=6
Bind Time 1.16s (± 6.63%) 1.22s (± 1.61%) ~ 1.18s 1.23s p=0.063 n=6
Check Time 22.14s (± 0.30%) 20.62s (± 0.44%) 🟩-1.52s (- 6.84%) 20.52s 20.73s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 26.49s (± 0.46%) 25.11s (± 0.36%) 🟩-1.37s (- 5.18%) 24.99s 25.20s p=0.005 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,654,303k (± 0.00%) 1,761,800k (± 0.00%) 🔻+107,497k (+ 6.50%) 1,761,758k 1,761,873k p=0.005 n=6
Parse Time 9.62s (± 0.38%) 9.85s (± 0.57%) +0.22s (+ 2.34%) 9.79s 9.95s p=0.005 n=6
Bind Time 2.97s (± 0.66%) 2.83s (± 0.65%) 🟩-0.14s (- 4.55%) 2.81s 2.85s p=0.005 n=6
Check Time 91.22s (± 1.10%) 82.52s (± 0.71%) 🟩-8.70s (- 9.53%) 81.75s 83.17s p=0.005 n=6
Emit Time 0.19s (± 4.29%) 0.20s (± 3.79%) ~ 0.19s 0.21s p=0.432 n=6
Total Time 104.00s (± 0.98%) 95.40s (± 0.63%) 🟩-8.60s (- 8.27%) 94.57s 96.05s p=0.005 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,366,974k (± 0.14%) 2,510,448k (± 0.02%) 🔻+143,474k (+ 6.06%) 2,509,934k 2,511,035k p=0.005 n=6
Parse Time 6.17s (± 0.73%) 6.26s (± 1.20%) +0.09s (+ 1.49%) 6.17s 6.34s p=0.045 n=6
Bind Time 2.13s (± 0.92%) 2.07s (± 1.00%) -0.06s (- 2.90%) 2.04s 2.09s p=0.005 n=6
Check Time 43.04s (± 0.51%) 40.48s (± 0.27%) 🟩-2.56s (- 5.96%) 40.34s 40.64s p=0.005 n=6
Emit Time 3.56s (± 8.31%) 3.18s (± 1.20%) 🟩-0.38s (-10.76%) 3.13s 3.24s p=0.008 n=6
Total Time 54.91s (± 0.66%) 51.98s (± 0.27%) 🟩-2.92s (- 5.32%) 51.83s 52.22s p=0.005 n=6
self-compiler - node (v18.15.0, x64)
Memory used 406,636k (± 0.01%) 425,786k (± 0.01%) 🔻+19,151k (+ 4.71%) 425,738k 425,822k p=0.005 n=6
Parse Time 2.67s (± 0.77%) 2.84s (± 0.47%) 🔻+0.17s (+ 6.23%) 2.82s 2.86s p=0.005 n=6
Bind Time 1.33s (± 0.63%) 1.08s (± 0.38%) 🟩-0.25s (-18.62%) 1.07s 1.08s p=0.003 n=6
Check Time 16.52s (± 0.63%) 15.42s (± 0.44%) 🟩-1.10s (- 6.68%) 15.33s 15.51s p=0.005 n=6
Emit Time 1.15s (± 1.31%) 1.13s (± 1.21%) -0.02s (- 1.74%) 1.11s 1.15s p=0.040 n=6
Total Time 21.67s (± 0.48%) 20.46s (± 0.37%) 🟩-1.21s (- 5.57%) 20.36s 20.55s p=0.005 n=6
vscode - node (v18.15.0, x64)
Memory used 2,760,260k (± 0.00%) 2,930,540k (± 0.00%) 🔻+170,280k (+ 6.17%) 2,930,498k 2,930,611k p=0.005 n=6
Parse Time 10.99s (± 0.39%) 11.12s (± 0.64%) +0.13s (+ 1.17%) 11.02s 11.21s p=0.013 n=6
Bind Time 3.61s (± 0.37%) 3.49s (± 0.16%) 🟩-0.12s (- 3.23%) 3.49s 3.50s p=0.004 n=6
Check Time 68.53s (± 0.34%) 62.26s (± 0.40%) 🟩-6.27s (- 9.14%) 61.91s 62.57s p=0.005 n=6
Emit Time 18.26s (± 0.35%) 16.62s (± 0.85%) 🟩-1.64s (- 8.98%) 16.45s 16.83s p=0.005 n=6
Total Time 101.39s (± 0.27%) 93.49s (± 0.44%) 🟩-7.90s (- 7.79%) 92.97s 94.11s p=0.005 n=6
webpack - node (v18.15.0, x64)
Memory used 396,897k (± 0.03%) 419,139k (± 0.02%) 🔻+22,241k (+ 5.60%) 419,062k 419,241k p=0.005 n=6
Parse Time 3.19s (± 2.45%) 3.34s (± 0.76%) 🔻+0.15s (+ 4.76%) 3.32s 3.39s p=0.014 n=6
Bind Time 1.46s (± 0.87%) 1.39s (± 0.98%) 🟩-0.07s (- 5.02%) 1.37s 1.40s p=0.005 n=6
Check Time 15.37s (± 0.23%) 14.45s (± 0.38%) 🟩-0.92s (- 5.95%) 14.39s 14.54s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.02s (± 0.43%) 19.18s (± 0.28%) 🟩-0.84s (- 4.17%) 19.12s 19.24s p=0.005 n=6
xstate - node (v18.15.0, x64)
Memory used 494,503k (± 0.01%) 537,535k (± 0.02%) 🔻+43,032k (+ 8.70%) 537,399k 537,717k p=0.005 n=6
Parse Time 4.91s (± 0.60%) 4.96s (± 0.42%) +0.04s (+ 0.81%) 4.93s 4.98s p=0.029 n=6
Bind Time 2.25s (± 0.95%) 2.21s (± 0.55%) -0.04s (- 1.56%) 2.20s 2.23s p=0.010 n=6
Check Time 4.69s (± 0.57%) 4.43s (± 1.00%) 🟩-0.26s (- 5.54%) 4.38s 4.50s p=0.005 n=6
Emit Time 0.11s (± 0.00%) 0.10s (± 4.01%) 🟩-0.01s (- 7.58%) 0.10s 0.11s p=0.007 n=6
Total Time 11.96s (± 0.29%) 11.71s (± 0.49%) -0.25s (- 2.12%) 11.66s 11.81s p=0.005 n=6

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,556ms (± 0.34%) 3,532ms (± 0.52%) -25ms (- 0.70%) 3,511ms 3,563ms p=0.045 n=6
Req 2 - geterr 8,310ms (± 0.37%) 7,691ms (± 2.17%) 🟩-618ms (- 7.44%) 7,597ms 8,029ms p=0.005 n=6
Req 3 - references 476ms (± 0.91%) 428ms (± 1.93%) 🟩-48ms (-10.08%) 421ms 440ms p=0.005 n=6
Req 4 - navto 406ms (± 0.59%) 343ms (± 0.80%) 🟩-63ms (-15.61%) 337ms 344ms p=0.004 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 132ms (± 5.18%) 121ms (± 8.64%) 🟩-12ms (- 8.94%) 109ms 138ms p=0.045 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,223ms (±11.83%) 3,065ms (± 9.62%) ~ 2,899ms 3,665ms p=0.092 n=6
Req 2 - geterr 5,515ms (±11.66%) 4,782ms (± 8.77%) 🟩-733ms (-13.29%) 4,571ms 5,637ms p=0.020 n=6
Req 3 - references 450ms (± 8.57%) 419ms (± 9.57%) ~ 366ms 448ms p=0.810 n=6
Req 4 - navto 443ms (± 1.57%) 320ms (± 8.48%) 🟩-123ms (-27.79%) 278ms 342ms p=0.005 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 109ms (± 1.13%) 95ms (±10.14%) 🟩-14ms (-12.60%) 82ms 102ms p=0.004 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,635ms (± 0.43%) 2,589ms (± 0.33%) -47ms (- 1.77%) 2,575ms 2,598ms p=0.005 n=6
Req 2 - geterr 1,753ms (± 1.84%) 1,545ms (± 0.66%) 🟩-208ms (-11.87%) 1,535ms 1,558ms p=0.005 n=6
Req 3 - references 115ms (± 9.25%) 94ms (± 0.80%) 🟩-21ms (-17.88%) 93ms 95ms p=0.005 n=6
Req 4 - navto 368ms (± 0.32%) 356ms (± 1.30%) 🟩-12ms (- 3.26%) 351ms 363ms p=0.005 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 311ms (± 1.69%) 290ms (± 1.35%) 🟩-21ms (- 6.66%) 284ms 296ms p=0.005 n=6

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 228.18ms (± 0.17%) 228.13ms (± 0.14%) ~ 226.56ms 230.84ms p=0.660 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 227.66ms (± 0.15%) 227.55ms (± 0.19%) -0.10ms (- 0.05%) 226.04ms 234.01ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 330.90ms (± 0.30%) 330.90ms (± 0.29%) ~ 322.58ms 339.22ms p=0.839 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 331.13ms (± 0.32%) 330.93ms (± 0.29%) -0.20ms (- 0.06%) 322.95ms 340.40ms p=0.005 n=600

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

weswigham

@jakebailey

The _children change appears to have even more greatly improved tsserver performance. Awesome.

rbuckton

@jakebailey jakebailey changed the base branch from jabaile/tsc-services to main

April 4, 2024 18:24

@typescript-bot

@jakebailey

@typescript-bot

@jakebailey

@typescript-bot

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
pack this ✅ Started ✅ Results

@typescript-bot

@typescript-bot

Hey @jakebailey, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/160982/artifacts?artifactName=tgz&fileId=B08E5D89B7EC46EFFAA2687CDAB8467544AC7B5B9F1217BCA7F8AEA8941E259E02&fileName=/typescript-5.5.0-insiders.20240404.tgz"
    }
}

and then running npm install.

There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/pr-build@5.5.0-pr-58045-49".;

@typescript-bot

@jakebailey Here are the results of running the user tests comparing main and refs/pull/58045/merge:

Everything looks good!

@typescript-bot

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,661k (± 0.02%) 295,654k (± 0.01%) ~ 295,616k 295,680k p=0.936 n=6
Parse Time 3.93s (± 0.41%) 3.93s (± 0.39%) ~ 3.91s 3.95s p=0.739 n=6
Bind Time 1.23s (± 1.49%) 1.23s (± 0.95%) ~ 1.22s 1.25s p=0.731 n=6
Check Time 11.99s (± 0.36%) 12.05s (± 0.16%) +0.06s (+ 0.47%) 12.03s 12.08s p=0.013 n=6
Emit Time 10.47s (± 0.36%) 10.47s (± 0.20%) ~ 10.45s 10.51s p=0.809 n=6
Total Time 27.62s (± 0.15%) 27.68s (± 0.14%) +0.06s (+ 0.21%) 27.61s 27.72s p=0.037 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 193,950k (± 1.02%) 192,748k (± 0.74%) ~ 192,112k 195,649k p=0.575 n=6
Parse Time 2.02s (± 0.93%) 2.00s (± 1.72%) ~ 1.95s 2.05s p=0.687 n=6
Bind Time 1.08s (± 0.95%) 1.08s (± 0.59%) ~ 1.07s 1.09s p=0.443 n=6
Check Time 13.92s (± 0.30%) 13.88s (± 0.30%) ~ 13.83s 13.94s p=0.225 n=6
Emit Time 3.84s (± 0.45%) 3.85s (± 0.85%) ~ 3.81s 3.89s p=0.936 n=6
Total Time 20.86s (± 0.22%) 20.81s (± 0.37%) ~ 20.74s 20.93s p=0.294 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,539k (± 0.01%) 347,541k (± 0.01%) ~ 347,527k 347,573k p=1.000 n=6
Parse Time 3.01s (± 0.93%) 2.97s (± 0.84%) ~ 2.94s 3.01s p=0.062 n=6
Bind Time 1.12s (± 0.46%) 1.11s (± 0.46%) ~ 1.11s 1.12s p=0.311 n=6
Check Time 8.26s (± 0.45%) 8.27s (± 0.25%) ~ 8.23s 8.29s p=0.629 n=6
Emit Time 4.87s (± 0.64%) 4.84s (± 0.35%) ~ 4.82s 4.86s p=0.146 n=6
Total Time 17.25s (± 0.40%) 17.19s (± 0.19%) ~ 17.16s 17.25s p=0.126 n=6
TFS - node (v18.15.0, x64)
Memory used 302,832k (± 0.00%) 302,852k (± 0.01%) ~ 302,817k 302,876k p=0.148 n=6
Parse Time 2.96s (± 0.87%) 2.96s (± 0.70%) ~ 2.93s 2.99s p=0.806 n=6
Bind Time 1.48s (± 0.28%) 1.48s (± 0.82%) ~ 1.47s 1.50s p=0.116 n=6
Check Time 9.25s (± 0.23%) 9.26s (± 0.33%) ~ 9.22s 9.31s p=0.627 n=6
Emit Time 5.31s (± 0.31%) 5.31s (± 0.60%) ~ 5.28s 5.36s p=0.570 n=6
Total Time 19.00s (± 0.21%) 19.01s (± 0.18%) ~ 18.96s 19.05s p=0.872 n=6
material-ui - node (v18.15.0, x64)
Memory used 510,182k (± 0.01%) 510,238k (± 0.00%) ~ 510,208k 510,264k p=0.078 n=6
Parse Time 3.94s (± 0.30%) 3.95s (± 0.28%) ~ 3.94s 3.96s p=0.354 n=6
Bind Time 1.46s (± 0.71%) 1.47s (± 0.82%) ~ 1.45s 1.48s p=0.210 n=6
Check Time 25.46s (± 0.25%) 25.40s (± 0.36%) ~ 25.29s 25.53s p=0.226 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 30.86s (± 0.21%) 30.82s (± 0.30%) ~ 30.71s 30.96s p=0.520 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,744,929k (± 0.00%) 1,744,950k (± 0.00%) ~ 1,744,902k 1,745,029k p=0.378 n=6
Parse Time 9.65s (± 0.50%) 9.63s (± 0.59%) ~ 9.58s 9.74s p=0.419 n=6
Bind Time 3.49s (± 0.23%) 3.48s (± 0.63%) ~ 3.45s 3.51s p=0.461 n=6
Check Time 82.32s (± 0.39%) 82.24s (± 0.53%) ~ 81.89s 82.94s p=0.748 n=6
Emit Time 0.19s (± 2.13%) 0.19s (± 2.13%) ~ 0.19s 0.20s p=1.000 n=6
Total Time 95.64s (± 0.35%) 95.54s (± 0.50%) ~ 95.13s 96.34s p=0.471 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,402,045k (± 0.05%) 2,402,269k (± 0.03%) ~ 2,401,211k 2,402,807k p=0.689 n=6
Parse Time 7.48s (± 0.77%) 7.54s (± 0.83%) ~ 7.46s 7.62s p=0.173 n=6
Bind Time 2.79s (± 0.57%) 2.79s (± 0.88%) ~ 2.75s 2.82s p=1.000 n=6
Check Time 49.38s (± 0.18%) 49.19s (± 0.30%) -0.19s (- 0.38%) 49.02s 49.41s p=0.031 n=6
Emit Time 3.99s (± 3.27%) 3.90s (± 1.67%) -0.09s (- 2.38%) 3.86s 4.03s p=0.044 n=6
Total Time 63.64s (± 0.15%) 63.44s (± 0.22%) -0.20s (- 0.31%) 63.30s 63.66s p=0.045 n=6
self-compiler - node (v18.15.0, x64)
Memory used 416,584k (± 0.01%) 416,592k (± 0.01%) ~ 416,536k 416,654k p=1.000 n=6
Parse Time 4.11s (± 0.42%) 4.12s (± 0.40%) ~ 4.10s 4.14s p=0.684 n=6
Bind Time 1.59s (± 0.84%) 1.59s (± 0.62%) ~ 1.58s 1.60s p=0.727 n=6
Check Time 22.45s (± 0.49%) 22.36s (± 0.38%) ~ 22.27s 22.49s p=0.199 n=6
Emit Time 1.66s (± 2.72%) 1.68s (± 1.02%) ~ 1.66s 1.71s p=0.414 n=6
Total Time 29.81s (± 0.49%) 29.75s (± 0.32%) ~ 29.66s 29.91s p=0.334 n=6
vscode - node (v18.15.0, x64)
Memory used 2,907,964k (± 0.00%) 2,907,969k (± 0.00%) ~ 2,907,922k 2,908,007k p=0.810 n=6
Parse Time 10.84s (± 0.29%) 10.84s (± 0.15%) ~ 10.82s 10.86s p=0.808 n=6
Bind Time 3.49s (± 0.38%) 3.49s (± 0.26%) ~ 3.48s 3.50s p=0.932 n=6
Check Time 62.16s (± 0.46%) 62.22s (± 0.49%) ~ 61.79s 62.66s p=0.748 n=6
Emit Time 16.49s (± 0.47%) 16.43s (± 0.41%) ~ 16.32s 16.51s p=0.173 n=6
Total Time 92.98s (± 0.25%) 92.98s (± 0.26%) ~ 92.64s 93.31s p=1.000 n=6
webpack - node (v18.15.0, x64)
Memory used 408,998k (± 0.02%) 408,982k (± 0.03%) ~ 408,858k 409,216k p=0.575 n=6
Parse Time 3.89s (± 0.50%) 3.89s (± 0.66%) ~ 3.87s 3.94s p=0.871 n=6
Bind Time 1.69s (± 0.95%) 1.68s (± 0.75%) ~ 1.66s 1.69s p=0.735 n=6
Check Time 16.86s (± 0.32%) 16.83s (± 0.22%) ~ 16.76s 16.86s p=0.374 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.44s (± 0.14%) 22.40s (± 0.26%) ~ 22.29s 22.45s p=0.227 n=6
xstate - node (v18.15.0, x64)
Memory used 513,425k (± 0.01%) 513,449k (± 0.00%) ~ 513,418k 513,474k p=0.810 n=6
Parse Time 3.28s (± 0.31%) 3.29s (± 0.32%) ~ 3.27s 3.30s p=0.801 n=6
Bind Time 1.57s (± 0.48%) 1.57s (± 0.33%) ~ 1.57s 1.58s p=0.241 n=6
Check Time 2.91s (± 0.36%) 2.90s (± 0.77%) ~ 2.87s 2.92s p=0.405 n=6
Emit Time 0.07s (±11.12%) 0.07s (±11.12%) ~ 0.07s 0.09s p=1.000 n=6
Total Time 7.85s (± 0.15%) 7.83s (± 0.21%) ~ 7.81s 7.85s p=0.224 n=6

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,377ms (± 0.93%) 2,345ms (± 0.34%) -32ms (- 1.34%) 2,333ms 2,355ms p=0.008 n=6
Req 2 - geterr 5,639ms (± 0.34%) 5,150ms (± 0.59%) 🟩-490ms (- 8.69%) 5,117ms 5,206ms p=0.005 n=6
Req 3 - references 319ms (± 1.04%) 295ms (± 2.10%) 🟩-24ms (- 7.38%) 287ms 299ms p=0.004 n=6
Req 4 - navto 272ms (± 0.00%) 228ms (± 0.00%) 🟩-44ms (-16.18%) 228ms 228ms p=0.001 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 88ms (± 2.32%) 77ms (± 0.67%) 🟩-11ms (-11.95%) 77ms 78ms p=0.003 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,520ms (± 0.73%) 2,509ms (± 0.89%) ~ 2,476ms 2,537ms p=0.471 n=6
Req 2 - geterr 4,159ms (± 0.20%) 3,900ms (± 1.53%) 🟩-259ms (- 6.23%) 3,863ms 4,020ms p=0.005 n=6
Req 3 - references 333ms (± 0.36%) 298ms (± 0.87%) 🟩-35ms (-10.52%) 294ms 301ms p=0.005 n=6
Req 4 - navto 298ms (± 1.07%) 237ms (± 3.54%) 🟩-61ms (-20.57%) 226ms 243ms p=0.004 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 75ms (± 5.90%) 70ms (± 3.06%) 🟩-5ms (- 6.89%) 68ms 74ms p=0.032 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,896ms (± 0.37%) 3,809ms (± 0.30%) -88ms (- 2.25%) 3,788ms 3,820ms p=0.005 n=6
Req 2 - geterr 2,450ms (± 0.98%) 2,279ms (± 1.02%) 🟩-171ms (- 6.97%) 2,255ms 2,314ms p=0.005 n=6
Req 3 - references 149ms (± 1.02%) 134ms (± 1.54%) 🟩-14ms (- 9.54%) 131ms 136ms p=0.005 n=6
Req 4 - navto 561ms (± 1.30%) 524ms (± 0.58%) 🟩-37ms (- 6.51%) 520ms 529ms p=0.005 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 458ms (± 1.99%) 442ms (± 0.75%) 🟩-16ms (- 3.46%) 436ms 446ms p=0.005 n=6

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 153.94ms (± 0.17%) 153.78ms (± 0.18%) -0.16ms (- 0.11%) 152.76ms 157.53ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 228.11ms (± 0.14%) 228.14ms (± 0.18%) ~ 226.50ms 233.04ms p=0.942 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 223.40ms (± 0.14%) 223.43ms (± 0.16%) ~ 221.96ms 228.18ms p=0.740 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 223.20ms (± 0.16%) 223.37ms (± 0.18%) +0.17ms (+ 0.08%) 221.68ms 229.09ms p=0.000 n=600

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey

The above looks exactly as expected for this PR targeting main; no change to tsc (it doesn't use this code), and all benefit for tsserver.

@typescript-bot

@jakebailey Here are the results of running the top 400 repos comparing main and refs/pull/58045/merge:

Everything looks good!

@jakebailey jakebailey changed the titleMonomorphize services allocators, just like the compiler Monomorphize allocators for tsserver/public API, just like core compiler

Apr 4, 2024

@jakebailey

@jakebailey

The benchmarking system now includes a benchmark for simulating tsc via the public API; testing:

@typescript-bot perf test this

@typescript-bot

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,691k (± 0.01%) 295,683k (± 0.01%) ~ 295,657k 295,732k p=1.000 n=6
Parse Time 3.93s (± 0.21%) 3.94s (± 0.41%) ~ 3.92s 3.96s p=0.392 n=6
Bind Time 1.23s (± 0.84%) 1.23s (± 0.51%) ~ 1.22s 1.24s p=0.654 n=6
Check Time 12.04s (± 0.17%) 12.02s (± 0.28%) ~ 11.98s 12.06s p=0.513 n=6
Emit Time 10.46s (± 0.34%) 10.46s (± 0.44%) ~ 10.41s 10.53s p=1.000 n=6
Total Time 27.65s (± 0.09%) 27.64s (± 0.28%) ~ 27.56s 27.74s p=0.573 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 192,766k (± 0.76%) 193,374k (± 0.92%) ~ 192,059k 195,701k p=0.689 n=6
Parse Time 2.04s (± 1.05%) 2.02s (± 1.50%) ~ 1.98s 2.07s p=0.139 n=6
Bind Time 1.08s (± 0.59%) 1.06s (± 0.49%) -0.02s (- 1.54%) 1.06s 1.07s p=0.006 n=6
Check Time 13.89s (± 0.60%) 13.96s (± 0.49%) ~ 13.88s 14.06s p=0.199 n=6
Emit Time 3.85s (± 0.76%) 3.86s (± 0.45%) ~ 3.84s 3.89s p=0.565 n=6
Total Time 20.86s (± 0.37%) 20.91s (± 0.38%) ~ 20.83s 21.05s p=0.521 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,540k (± 0.01%) 347,538k (± 0.01%) ~ 347,504k 347,582k p=0.630 n=6
Parse Time 3.68s (± 1.47%) 3.69s (± 1.11%) ~ 3.63s 3.74s p=0.936 n=6
Bind Time 1.37s (± 1.10%) 1.39s (± 1.90%) ~ 1.37s 1.44s p=0.217 n=6
Check Time 10.17s (± 0.43%) 10.20s (± 0.46%) ~ 10.14s 10.27s p=0.258 n=6
Emit Time 6.01s (± 0.46%) 6.00s (± 0.42%) ~ 5.98s 6.05s p=0.806 n=6
Total Time 21.23s (± 0.31%) 21.28s (± 0.23%) ~ 21.22s 21.35s p=0.198 n=6
TFS - node (v18.15.0, x64)
Memory used 302,846k (± 0.01%) 302,835k (± 0.00%) ~ 302,821k 302,842k p=0.471 n=6
Parse Time 2.97s (± 0.72%) 2.97s (± 1.16%) ~ 2.94s 3.01s p=0.462 n=6
Bind Time 1.48s (± 0.35%) 1.47s (± 0.51%) ~ 1.46s 1.48s p=0.247 n=6
Check Time 9.26s (± 0.25%) 9.27s (± 0.26%) ~ 9.24s 9.31s p=0.627 n=6
Emit Time 5.30s (± 0.45%) 5.30s (± 0.94%) ~ 5.25s 5.38s p=1.000 n=6
Total Time 19.00s (± 0.21%) 19.01s (± 0.23%) ~ 18.95s 19.08s p=0.807 n=6
material-ui - node (v18.15.0, x64)
Memory used 510,160k (± 0.01%) 510,166k (± 0.01%) ~ 510,112k 510,278k p=1.000 n=6
Parse Time 3.92s (± 0.36%) 3.93s (± 0.30%) ~ 3.92s 3.95s p=0.357 n=6
Bind Time 1.45s (± 0.71%) 1.46s (± 0.80%) ~ 1.45s 1.48s p=0.054 n=6
Check Time 25.48s (± 0.21%) 25.43s (± 0.43%) ~ 25.30s 25.57s p=0.689 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 30.84s (± 0.17%) 30.82s (± 0.37%) ~ 30.67s 30.95s p=1.000 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,744,935k (± 0.00%) 1,744,950k (± 0.00%) ~ 1,744,913k 1,745,007k p=0.689 n=6
Parse Time 9.59s (± 0.41%) 9.64s (± 0.50%) ~ 9.59s 9.72s p=0.149 n=6
Bind Time 3.45s (± 0.70%) 3.45s (± 1.08%) ~ 3.39s 3.49s p=0.685 n=6
Check Time 81.68s (± 0.30%) 82.20s (± 0.34%) +0.52s (+ 0.63%) 81.70s 82.48s p=0.025 n=6
Emit Time 0.19s (± 2.13%) 0.19s (± 2.67%) ~ 0.19s 0.20s p=0.595 n=6
Total Time 94.91s (± 0.28%) 95.48s (± 0.27%) +0.57s (+ 0.60%) 95.06s 95.78s p=0.013 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,296,646k (± 0.03%) 2,296,712k (± 0.03%) ~ 2,296,068k 2,297,638k p=1.000 n=6
Parse Time 7.42s (± 0.91%) 7.43s (± 1.15%) ~ 7.30s 7.54s p=0.810 n=6
Bind Time 2.78s (± 0.95%) 2.82s (± 1.08%) +0.04s (+ 1.44%) 2.78s 2.87s p=0.045 n=6
Check Time 49.34s (± 0.19%) 49.40s (± 0.25%) ~ 49.20s 49.55s p=0.298 n=6
Emit Time 3.98s (± 0.63%) 4.04s (± 3.47%) ~ 3.92s 4.29s p=1.000 n=6
Total Time 63.50s (± 0.21%) 63.70s (± 0.34%) ~ 63.35s 63.98s p=0.093 n=6
self-build-src-public-api - node (v18.15.0, x64)
Memory used 2,257,958k (± 0.03%) 2,371,367k (± 0.02%) 🔻+113,409k (+ 5.02%) 2,370,644k 2,372,032k p=0.005 n=6
Parse Time 5.08s (± 1.65%) 5.17s (± 0.49%) ~ 5.14s 5.20s p=0.093 n=6
Bind Time 1.80s (± 1.04%) 1.70s (± 1.22%) 🟩-0.10s (- 5.66%) 1.67s 1.72s p=0.005 n=6
Check Time 36.50s (± 0.40%) 34.36s (± 0.27%) 🟩-2.14s (- 5.87%) 34.27s 34.50s p=0.005 n=6
Emit Time 2.93s (± 1.41%) 2.69s (± 0.94%) 🟩-0.24s (- 8.09%) 2.66s 2.72s p=0.005 n=6
Total Time 46.32s (± 0.32%) 43.94s (± 0.17%) 🟩-2.38s (- 5.13%) 43.84s 44.05s p=0.005 n=6
self-compiler - node (v18.15.0, x64)
Memory used 419,494k (± 0.01%) 419,506k (± 0.01%) ~ 419,464k 419,559k p=0.575 n=6
Parse Time 4.11s (± 0.71%) 4.11s (± 0.50%) ~ 4.08s 4.14s p=0.935 n=6
Bind Time 1.59s (± 0.62%) 1.59s (± 0.62%) ~ 1.58s 1.61s p=0.933 n=6
Check Time 22.34s (± 0.31%) 22.52s (± 0.38%) +0.18s (+ 0.79%) 22.38s 22.62s p=0.016 n=6
Emit Time 1.72s (± 1.63%) 1.73s (± 1.34%) ~ 1.70s 1.77s p=0.462 n=6
Total Time 29.76s (± 0.37%) 29.95s (± 0.28%) +0.19s (+ 0.66%) 29.80s 30.01s p=0.013 n=6
vscode - node (v18.15.0, x64)
Memory used 2,908,000k (± 0.00%) 2,907,952k (± 0.00%) ~ 2,907,786k 2,908,081k p=0.471 n=6
Parse Time 12.93s (± 0.34%) 12.95s (± 0.35%) ~ 12.89s 13.01s p=0.571 n=6
Bind Time 4.15s (± 0.34%) 4.16s (± 0.33%) ~ 4.14s 4.18s p=0.677 n=6
Check Time 72.33s (± 0.50%) 72.48s (± 0.78%) ~ 71.60s 73.21s p=0.521 n=6
Emit Time 20.10s (± 8.14%) 21.38s (±10.21%) ~ 19.24s 23.68s p=0.688 n=6
Total Time 109.50s (± 1.70%) 110.95s (± 2.43%) ~ 107.90s 113.45s p=0.378 n=6
webpack - node (v18.15.0, x64)
Memory used 409,046k (± 0.01%) 409,055k (± 0.02%) ~ 408,908k 409,154k p=0.810 n=6
Parse Time 4.81s (± 0.74%) 4.80s (± 0.50%) ~ 4.77s 4.84s p=0.294 n=6
Bind Time 2.06s (± 0.87%) 2.06s (± 1.24%) ~ 2.02s 2.08s p=1.000 n=6
Check Time 20.96s (± 0.44%) 20.87s (± 0.33%) -0.09s (- 0.41%) 20.79s 21.00s p=0.037 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 27.84s (± 0.33%) 27.73s (± 0.23%) ~ 27.64s 27.80s p=0.092 n=6
xstate - node (v18.15.0, x64)
Memory used 513,675k (± 0.02%) 513,641k (± 0.02%) ~ 513,491k 513,746k p=0.471 n=6
Parse Time 4.90s (± 0.97%) 4.92s (± 0.88%) ~ 4.87s 4.98s p=0.423 n=6
Bind Time 2.29s (± 0.71%) 2.31s (± 1.32%) ~ 2.28s 2.36s p=0.332 n=6
Check Time 4.32s (± 0.52%) 4.30s (± 0.90%) ~ 4.26s 4.37s p=0.170 n=6
Emit Time 0.11s (± 0.00%) 0.11s (± 0.00%) ~ 0.11s 0.11s p=1.000 n=6
Total Time 11.62s (± 0.49%) 11.63s (± 0.56%) ~ 11.56s 11.70s p=0.748 n=6

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,560ms (± 0.36%) 3,523ms (± 0.60%) -37ms (- 1.05%) 3,485ms 3,544ms p=0.010 n=6
Req 2 - geterr 8,318ms (± 0.80%) 7,611ms (± 0.72%) 🟩-707ms (- 8.49%) 7,566ms 7,711ms p=0.005 n=6
Req 3 - references 475ms (± 0.96%) 431ms (± 0.50%) 🟩-44ms (- 9.24%) 428ms 433ms p=0.005 n=6
Req 4 - navto 407ms (± 0.22%) 341ms (± 0.78%) 🟩-66ms (-16.30%) 336ms 343ms p=0.005 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 131ms (± 0.79%) 114ms (± 1.47%) 🟩-17ms (-12.76%) 111ms 115ms p=0.004 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,727ms (± 0.73%) 3,669ms (± 1.21%) ~ 3,620ms 3,737ms p=0.066 n=6
Req 2 - geterr 6,144ms (± 1.37%) 5,635ms (± 0.76%) 🟩-509ms (- 8.28%) 5,585ms 5,711ms p=0.005 n=6
Req 3 - references 492ms (± 0.63%) 443ms (± 0.67%) 🟩-49ms (- 9.90%) 440ms 448ms p=0.005 n=6
Req 4 - navto 445ms (± 0.95%) 341ms (± 3.17%) 🟩-104ms (-23.28%) 336ms 363ms p=0.005 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 110ms (± 3.03%) 102ms (± 0.82%) 🟩-8ms (- 7.45%) 100ms 102ms p=0.004 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,905ms (± 1.00%) 3,819ms (± 0.58%) -86ms (- 2.20%) 3,790ms 3,850ms p=0.005 n=6
Req 2 - geterr 2,465ms (± 1.44%) 2,268ms (± 0.48%) 🟩-197ms (- 7.99%) 2,251ms 2,282ms p=0.005 n=6
Req 3 - references 144ms (± 1.73%) 131ms (± 1.73%) 🟩-14ms (- 9.58%) 128ms 133ms p=0.005 n=6
Req 4 - navto 557ms (± 1.07%) 531ms (± 1.37%) 🟩-26ms (- 4.69%) 523ms 543ms p=0.005 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 453ms (± 0.56%) 439ms (± 1.27%) 🟩-14ms (- 3.02%) 432ms 446ms p=0.005 n=6

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr

Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 222.77ms (± 0.16%) 222.61ms (± 0.18%) -0.16ms (- 0.07%) 220.62ms 228.04ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 228.65ms (± 0.16%) 228.64ms (± 0.19%) ~ 227.01ms 239.29ms p=0.338 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 273.78ms (± 0.30%) 273.90ms (± 0.31%) ~ 266.35ms 283.77ms p=0.076 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 332.39ms (± 0.28%) 332.32ms (± 0.29%) ~ 323.85ms 337.56ms p=0.393 n=600

System info unknown

Hosts

Scenarios

Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey

Now that the benchmarker can test the public API as tsc:

image

This was referenced

Jul 2, 2024