Re-organize build-dir by package + hash, rather than artifact type (original) (raw)

Implementation:

Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#build-dir-new-layout

Resolves

Shadowing binaries on Windows (Workspace binary artifacts are added to target/debug/deps on Windows #7919)
Collisions between intermediate build artifacts (Output filename collision with dylib on windows release mode #8794)
Inconsistent use of -Cextra-filename (Consider changing hashing behavior for executables. #8332)

Past efforts

Unify build directories. #6668

Overview

.fingerprint, deps/, build-script, and build-script-execution/ move under build/<pkgname>/<unit-id>
- -L params to rustc were updated
- cargo clean -p was updated
- incremental/ is untouched
-Cextra-filename is removed from all bins (fix(layout): Remove hashes from bins in new layout #16351)

Known issues

As a side effect, we pass a lot more parameters to rustc, likely making cargo -vv more annoying, similar to Reduce --verbose output of --check-cfg args #13941
It also makes impossible to just copy paste commands into your shell if the limit is exceeded. See bjorn3's comment.
This will break people relying on build-dir internals
- See #t-cargo > build-dir tool compatibility @ 💬
Increases the need for solutions for when people rely on build-dir internals
- Access to compiler artifact notifications messages for custom --emit flags #13672
- cargo ./target fills with outdated artifacts as toolchains are updated/changed #5026
- Allow build scripts to stage final artifacts #13663
- RFC: cargo-sbom rfcs#3553 (more precise determination of what is built than cargo metadata)
cargo clean -p only cleans up files for the current layout rather than supporting both

Unresolved issues

Open questions

Transition plan: while build-dir isn't stable, enough tools rely on the layout that we'd want to setup a transition plan so they can have time to test against the new layout and work to support both
What do we call the directory? I said build/ as its all encompassing
- #t-cargo > directory naming with new build layout @ 💬
Can the old build/ and deps/ content live in the same place?
- Seems to work
How should we handle incremental/?
- rustc loads incremental artifacts only for local crates. cross-crates is from rmeta. Therefore per-crate incremental artifacts should be fine.
- Given incremental directory has its own flock mechanism, we don't need to add flock for that directory.
- See #t-compiler > ✔ Cargo switching to one `-C incremental` directory per c...
Under the new layout, should cargo clean -p also cleans old layout paths? See feat(build-dir): Reorganize build-dir layout #15947 (comment)
Is the current handling of build scripts sufficient or should we explicitly split them into separate entries. We separated build script build and execution

Future extensions

Deferred layout changes

Can we remove -Cextra-filename completely as it is redundant with the build-unit's <name>/<hash>
- It may not be possible to remove -Cextra-filename for rlibs. rustc relies on it. See feat(build-dir): Reorganize build-dir layout #15947 (comment)
Can we remove <tuple> from the parent path?
- In addition to reducing path length pressure on Windows, could allow sharing of host builds between platforms
- Blocked on More granular locking in cargo_rustc #4282
Can we remove <profile> from the parent path?
- In addition to reducing path length pressure on Windows, could allow sharing when build units have the same settings across profiles
- <hash> is the -C extra-filename hash and doesn't encompass all of fingerprinting, so we'd need to audit if there are cases that don't change the hash that we'd still need per-profile
- Changing of local source is one example, so at least local packages still need to be scoped by profile
- Blocked on More granular locking in cargo_rustc #4282
Can we simplify how fingerprints are stored, reducing pressure on path lengths
If Windows filesystem perf issues crop up with <pkgname>/<hash>, we can go with prefixes to reduce the number of items within a directory. See bjorn3's comment.
- if deps/ hasn't been a problem, this is unlikely to be a problem

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Original

note: I specify build-dir to clarify which half of #14125 I'm referring to. The files and layout of build-dir does not have compatibility guarantees (source).

Currently, build-dir is laid out like

target/
- <target-platform>/?
  * <profile>/
  * incremental/
  * build/
  * <package>-<hash>/
  * build*script-*
  * deps/
  * <package>-<hash>*

Currently,

cargo clean -p <package> will operate on everything for <package>

In the future, we could have

GC will track and operate on everything for <package>-<hash> (cargo ./target fills with outdated artifacts as toolchains are updated/changed #5026)
Change the locking so only overlapping <package>-<hash> that are being built block (More granular locking in cargo_rustc #4282)
Centrally cache <package>-<hash> artifacts across all projects (Per-user compiled artifact cache #5931)
Reduced overhead on systems that are slow when there are a lot of files within a directory (see also Adding tests has a high friction #15691)

These could be aided by re-arranging the build-dir to be organized around <package>-<hash>, like

target/
- <target-platform>/?
  * <profile>/
  * incremental/
  * build/
  * <package>-<hash>/
  * build*script-*
  * *.d

Side effects

We'll have to change how we invoke rustc which will increase the length of the command-line
- Currently, we blindly point rustc at deps/ and rustc finds the files it needs. We'll instead need to point to each individual artifact rustc may need.