Redefine CARGO_TARGET_DIR to be only an artifacts directory · Issue #14125 · rust-lang/cargo (original) (raw)

For design, see #14125 (comment)

For tracking the artifact-dir side of this, see #6790

Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#build-dir

Testing instructions: #14125 (comment)

Implementation

Stabilizing this would also resolve

Known issues:

Decisions

artifact-dir files:

build-dir files (intermediate artifacts, build state, caches):

Open questions

Deferred

Original Issue:

Problem

There are a couple of issues with the CARGO_TARGET_DIR that are seemingly in conflict with each other:

  1. Multiple locations of target dirs complicate excluding them from backups and full-disk search, cleanup of the temp files, moving temp files to dedicated partitions, out of slow network drives or container mounts, etc. Users don't like that the target dir is huge, and multiple instances of it add up to lot of disk space. Users would prefer a central location to ease management of the temp files, and also to dedupe/reuse dependencies across many projects.
  2. People (and tools) are relying on a relative ./target directory being present to copy or run built files out of there. Additionally, users may not want to configure a shared CARGO_TARGET_DIR due to risk of file name conflicts between projects.

However, the dilemma between 1 and 2 exists only because Cargo uses CARGO_TARGET_DIR for two different roles:

  1. A cache for all intermediate build products (a place where crates.io crates are built, where compiler-private temp files are) which aren't project-specific, and/or files that users don't need to access directly.
  2. A location for user-facing final build products (artifacts) that users expect to be there and need to access.

Proposed Solution

So to satisfy both uses, I suggest to change the thinking about what the role of CARGO_TARGET_DIR should be. Instead of thinking where to put the same huge all-purpose mixed CARGO_TARGET_DIR, think how to deduplicate and slim CARGO_TARGET_DIR, and move everything non-user-facing out of it.

Instead of merging or sharding the CARGO_TARGET_DIR as-is with all of its current content, and adding --artifact-dir as a separate place where final products are being copied to — make CARGO_TARGET_DIR to be the artifact dir (without copying).

As long as the CARGO_TARGET_DIR dir is the place for all of the build files, of all crates including all the crates.io and local builds, with all the caches, all the temp junk, then this is going to be a problematic large directory that needs to be managed. But if the purpose of the ./target dir was changed to be only for user-facing files (files that users can name, and would access via ./target path themselves), then this directory would be relatively small, with a good reason to stay workspace-relative.

What isn't an intermediate build product? (and should stay in ./target)

So generally files that users build intentionally, and may want to access directly (run themselves, or package up for distribution) and files that users may need configure their IDE and debugger to find inside the project.

Crates in [patch.crates-io] with a path are a gray area, an might also have their artifacts included in the ./target dir (but in some way that avoids clobbering workspaces' files).

What isn't a final build product, and doesn't belong to ./target:

All of these should be built in some other shared build cache dir (one that is not inside CARGO_TARGET_DIR), configurable by a new option/env var.

Registry dependencies would get unique paths derived from rustc version + package IDs + enabled features (so that different crates using different features don't invalidate each others' caches all the time). This would enable sharing built crates.io dependencies across all projects for the same local user, without also causing local workspaces to clobber each others' CARGO_TARGET_DIR/profile/product paths. Temp directories for local projects would need some hashed paths in the shared build/temp dir too.

Advantages

Notes

No response