Stabilize Frontmatter by epage · Pull Request #148051 · rust-lang/rust (original) (raw)

Stabilization report

Summary

This report proposes the stabilization of #[feature(frontmatter)] in preparation for Cargo Script to be stabilized.

A frontmatter is a special kind of attribute intended to be read and modified by external tools, like Cargo. This puts a particular constraint on this attribute in that full awareness of the Rust grammar would be prohibitive.

For example:

#!/usr/bin/env cargo

[dependencies] clap = "4"

fn main() { }

The goal of Cargo script is to improve:

Learning by reducing the bar for experimenting and prototyping
Collaboration, including bug reports, chats, blogs, and books
Interoperability between tools that need similar solutions (playgrounds,
One off utilities where it makes sense (e.g. integrating with Rust-specific libraries)

Closes #136889

Tracking:

Reference PRs:

docs(ref): Specify frontmatter reference#1974

cc @rust-lang/lang @rust-lang/lang-advisors, @rust-lang/compiler

What is stabilized

This is stabilizing frontmatter, an optional section in a source file that can only exist before any Rust code or comments, for use by external tools.

#!/usr/bin/env cargo

[dependencies] clap = "4"

fn main() { }

After the opening frontmatter fence, an infostring is allowed that can identify the contents.
Calling tools are responsible for interpreting the infostring.

What isn't stabilized

Only a subset of the allowed infostring syntax is being stabilized.

RFC: "an optional term (one or more characters excluding whitespace and commas)"
rustc: (XID_Start | _) ( XID_Continue | -|. )*
The RFC allowed - in the start of an infostring which is ambiguous with frontmatter open which accepts 3 or more -
We simplified to a minimal, closed set as a precaution with a focus on what would make most file names work
See Implement RFC 3503: frontmatters #140035

Future possibilities include:

Infostring attributes: the supported syntax for infostrings was restricted to leave the door open
Tracking this as an attribute in the AST
Multiple frontmatters

Design

Reference

docs(ref): Specify frontmatter reference#1974

RFC history

RFC 3503 adds a new lexical section to the Reference to describe the frontmatter, including the grammar.

Answers to unresolved questions

What questions were left unresolved by the RFC? How have they been answered? Link to any relevant lang decisions.

The RFC has no unresolved questions

Post-RFC changes

What other user-visible changes have occurred since the RFC was accepted? Describe both changes that the lang team accepted (and link to those decisions) as well as changes that are being presented to the team for the first time in this stabilization report.

infostring: allowed characters (discussed under "What isn't stabilized")
whitespace:
- RFC: "whitespace"
- rustc: "horizontal whitespace" per the Unicode standard
- The intention when written was horizontal whitespace and I overlooked how broad rustc defines it. See also #145971
acceptable places for frontmatter
- RFC: "This applies anywhere shebang stripping is performed. For example, if include! strips shebangs, then it will also frontmatter."
- rustc: support removed from --cfg (Disallow frontmatter in --cfg and --check-cfg arguments #146137) and proc-macros (Strip frontmatter in fewer places #146340)
- rustc: support removed from include! in expression/statement-content (still supported in item context)
  * See Don't strip shebang in expr-ctxt include!(…) #146377 for having shebangs match
  * discussion at Stabilize Frontmatter #148051 (comment)
- It was generally unexpected that shebangs were supported in --cfg
- include! can be used to inject code into the middle of a file which becomes ambiguous between a frontmatter start and a very negated number (----x)
cap code fence -s to 255 like # in raw string literals
- see Stabilize Frontmatter #148051 (comment)

Key points

What decisions have been most difficult and what behaviors to be stabilized have proved most contentious? Summarize the major arguments on all sides and link to earlier documents and discussions.

whitespace between opening and infostring
- The RFC specified it
- Initially, Implement RFC 3503: frontmatters #140035 was going to remove it until the discussion at Implement RFC 3503: frontmatters #140035 (comment)
- Reasons against: "What benefit would allowing whitespace before the infostring give?"
- Reasons for, at that time: RFC said it and we're matching markdown
- Additional reasons for: style team has said that they want to specify the use of the space (see #145617).
rust-analyzer wanting an external indicator for cargo-scripts
- This would allow them to differentiate between unlinked files and cargo scripts without ambiguity and without having to read the content. The latter means they either pre-load and take a performance hit or load on-demand
- Resolving this runs counter to some of the design goals within the RFC
- From the outside, it is not clear why this is as big of a problem as they feel it is
- As this is re-litigating an RFC decision, the responsibility for making the case for why this is important was put on them and there has been no progress on this in the last 2 months
- See also #t-compiler/rust-analyzer > Frontmatter vs `import_rules!` for scripts @ 💬
whether indented --- needs to be escaped
- The RFC only said that --- at the start of a line needs escaping
- #140035 also had indented --- needing escaping
- #145754 switched the implementation to match the RFC
interactions with -Zunpretty
- questions raised at Add unstable frontmatter support #137193 (comment)
- went forward with passing them through like shebang at fix: Include frontmatter in -Zunpretty output #143708
whether external tools should silently fail, delegating error reporting to rustc
- Cargo has gone with high quality error reporting to report problems quickly, including non-compiling commands, rather than deferring errors to compilation. In particular, the contents of the frontmatter can make a big difference in how compilation happens.

Nightly extensions

Are there extensions to this feature that remain unstable? How do we know that we are not accidentally committing to those?

No nightly extensions exist

Doors closed

What doors does this stabilization close for later changes to the language? E.g., does this stabilization make any other RFCs, lang experiments, or known in-flight proposals more difficult or impossible to do later?

There are no known efforts that this affects.

Feedback

Call for testing

Has a "call for testing" been done? If so, what feedback was received?

Call for testing with feedback at:

Previous call for testing: rust-lang/rfcs#3424 (comment) with some feedback at rust-lang/cargo#12207 (comment)

Overall, feedback is enthusiastic. Any quibbles are with the Cargo side.

Nightly use

Do any known nightly users use this feature? Counting instances of #![feature(FEATURE_NAME)] on GitHub with grep might be informative.

Samples from grepping for -Zscript

Implementation

Major parts

Summarize the major parts of the implementation and provide links into the code and to relevant PRs.

See, e.g., this breakdown of the major parts of async closures:

https://rustc-dev-guide.rust-lang.org/coroutine-closures.html

Parts:

lexer
parser

PRs:

Coverage

Summarize the test coverage of this feature.

Consider what the "edges" of this feature are. We're particularly interested in seeing tests that assure us about exactly what nearby things we're not stabilizing. Tests should of course comprehensively demonstrate that the feature works. Think too about demonstrating the diagnostics seen when common mistakes are made and the feature is used incorrectly.

Within each test, include a comment at the top describing the purpose of the test and what set of invariants it intends to demonstrate. This is a great help to our review.

Describe any known or intentional gaps in test coverage.

Contextualize and link to test folders and individual tests.

Tests (directory):

Location
- May or may not be preceded by a shebang
- Can't have comments (or other non-whitespace content) before opening
- Whitespace is allowed before frontmatter
General
- CR/LR is supported
- Invalid to have multiple frontmatters
Infostring
- Can't have a , in the infostring
- Can't have a leading . or -
- Can have a . or - elsewhere
- Can be preceded by horizontal whitespace
Fences
- Invalid to have non-whitespace content after close
- Invalid to have whitespace between newline and fence
- Horizontal whitespace is allowed after fences
- Fence must be closed
  * On missing close, recover after use
- Open close lengths must match
Content
- Cover whitespace corner cases in content
- Multi-byte characters don't cause crashes
- Allow characters not accepted by the lexer
Escaping
- Non-leading --- doesn't need escaping
- More - escape lines starting with fewer -
Other uses
- include! doesn't treat --- as frontmatter
- proc-macro doesn't treat --- as frontmatter

Outstanding bugs

What outstanding bugs involve this feature? List them. Should any block the stabilization? Discuss why or why not.

No known bugs

Outstanding FIXMEs

What FIXMEs are still in the code for that feature and why is it OK to leave them there?

An idea for future editions

// FIXME(frontmatter): Consider stripping frontmatter in a future edition. We can't strip them
// in the current edition since that would be breaking.
// See also https://github.com/rust-lang/rust/issues/145520.
// Alternatively, stop stripping shebangs here, too, if T-lang and crater approve.
source_file_to_stream(psess, source_file, override_span, StripTokens::Shebang)

Diagnostic improvements can be iterated on over time, particularly with use feedback

---
//~^ ERROR: expected item, found `-`
// FIXME(frontmatter): make this diagnostic better
---

---
---

---
//~^ ERROR: expected item, found `-`
// FIXME(frontmatter): make this diagnostic better
---

Both can be handled as need is shown.

Tool changes

What changes must be made to our other tools to support this feature. Has this work been done? Link to any relevant PRs and issues.

rustfmt (non-blocking per https://rust-lang.zulipchat.com/#narrow/channel/357797-t-rustfmt/topic/Parsing.20of.20frontmatters/near/535570252)
- docs(style): Specify the frontmatter style #145617 (style guide)
- test(rustfmt): Verify frontmatter is preserved #145766 (frontmatter preserved)
- Style frontmatter according to the style guide rustfmt#6709 (formatting)
rust-analyzer
- feat(parser): Don't error on frontmatter rust-analyzer#20854 (syntax support)
- Add shebang support for "cargo script" rust-analyzer#15318 (attaching scripts)
rustdoc (both JSON and HTML)
- Open items present on doctests
cargo (dependent on this)
- Tracking Issue for cargo-script RFC 3424 cargo#12207
clippy
- just works
rustup
- N/A
docs.rs
- N/A
playground (non-blocking)
- Add cargo-script support rust-playground#1049

Breaking changes

If this stabilization represents a known breaking change, link to the crater report, the analysis of the crater report, and to all PRs we've made to ecosystem projects affected by this breakage. Discuss any limitations of what we're able to know about or to fix.

No known breaking change

Type system, opsem

Compile-time checks

What compilation-time checks are done that are needed to prevent undefined behavior?

Link to tests demonstrating that these checks are being done.

N/A, this is only relevant to tools inspecting the source

Type system rules

What type system rules are enforced for this feature and what is the purpose of each?

N/A, this is only relevant to tools inspecting the source