Make ast::TokenKind more like lexer::TokenKind by nnethercote · Pull Request #137902 · rust-lang/rust (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation3 Commits2 Checks6 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

@nnethercote

@nnethercote

BinOpToken is badly named, because it only covers the assignable binary ops and excludes comparisons and &&/||. Its use in ast::TokenKind does allow a small amount of code sharing, but it's a clumsy factoring.

This commit removes ast::TokenKind::BinOp{,Eq}, replacing each one with 10 individual variants. This makes ast::TokenKind more similar to rustc_lexer::TokenKind, which has individual variants for all operators.

Although the number of lines of code increases, the number of chars decreases due to the frequent use of shorter names like token::Plus instead of token::BinOp(BinOpToken::Plus).

@nnethercote

For consistency with rustc_lexer::TokenKind::Bang, and because other ast::TokenKind variants generally have syntactic names instead of semantic names (e.g. Star and DotDot instead of Mul and Range).

@rustbot rustbot added S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

T-compiler

Relevant to the compiler team, which will review and decide on the PR/issue.

T-rustdoc

Relevant to the rustdoc team, which will review and decide on the PR/issue.

labels

Mar 2, 2025

@rustbot

Some changes occurred in src/tools/rustfmt

cc @rust-lang/rustfmt

compiler-errors

@compiler-errors

r? compiler-errors @bors r+ rollup

@bors

📌 Commit 53167c0 has been approved by compiler-errors

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors

Status: Waiting on bors to run and complete tests. Bors will change the label on completion.

and removed S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

labels

Mar 3, 2025

bors added a commit to rust-lang-ci/rust that referenced this pull request

Mar 4, 2025

@bors

…iaskrgr

Rollup of 12 pull requests

Successful merges:

r? @ghost @rustbot modify labels: rollup

rust-timer added a commit to rust-lang-ci/rust that referenced this pull request

Mar 4, 2025

@rust-timer

github-actions bot pushed a commit to model-checking/verify-rust-std that referenced this pull request

Mar 14, 2025

@bors

…iaskrgr

Rollup of 12 pull requests

Successful merges:

r? @ghost @rustbot modify labels: rollup

flip1995 pushed a commit to flip1995/rust that referenced this pull request

Mar 20, 2025

@bors

…iaskrgr

Rollup of 12 pull requests

Successful merges:

r? @ghost @rustbot modify labels: rollup

nnethercote added a commit to nnethercote/rust that referenced this pull request

Apr 17, 2025

@nnethercote

By replacing them with {Open,Close}{Param,Brace,Bracket,Invisible}.

PR rust-lang#137902 made ast::TokenKind more like lexer::TokenKind by replacing the compound BinOp{,Eq}(BinOpToken) variants with fieldless variants Plus, Minus, Star, etc. This commit does a similar thing with delimiters. It also makes ast::TokenKind more similar to parser::TokenType.

This requires a few new methods:

Despite these additions, it's a net reduction in lines of code. This is because e.g. token::OpenParen is so much shorter than token::OpenDelim(Delimiter::Parenthesis) that many multi-line forms reduce to single line forms. And many places where the number of lines doesn't change are still easier to read, just because the names are shorter, e.g.:

-   } else if self.token != token::CloseDelim(Delimiter::Brace) {
+   } else if self.token != token::CloseBrace {

nnethercote added a commit to nnethercote/rust that referenced this pull request

Apr 17, 2025

@nnethercote

By replacing them with {Open,Close}{Param,Brace,Bracket,Invisible}.

PR rust-lang#137902 made ast::TokenKind more like lexer::TokenKind by replacing the compound BinOp{,Eq}(BinOpToken) variants with fieldless variants Plus, Minus, Star, etc. This commit does a similar thing with delimiters. It also makes ast::TokenKind more similar to parser::TokenType.

This requires a few new methods:

Despite these additions, it's a net reduction in lines of code. This is because e.g. token::OpenParen is so much shorter than token::OpenDelim(Delimiter::Parenthesis) that many multi-line forms reduce to single line forms. And many places where the number of lines doesn't change are still easier to read, just because the names are shorter, e.g.:

-   } else if self.token != token::CloseDelim(Delimiter::Brace) {
+   } else if self.token != token::CloseBrace {

GuillaumeGomez pushed a commit to GuillaumeGomez/rust that referenced this pull request

Apr 18, 2025

@bors

…iaskrgr

Rollup of 12 pull requests

Successful merges:

r? @ghost @rustbot modify labels: rollup

nnethercote added a commit to nnethercote/rust that referenced this pull request

Apr 20, 2025

@nnethercote

By replacing them with {Open,Close}{Param,Brace,Bracket,Invisible}.

PR rust-lang#137902 made ast::TokenKind more like lexer::TokenKind by replacing the compound BinOp{,Eq}(BinOpToken) variants with fieldless variants Plus, Minus, Star, etc. This commit does a similar thing with delimiters. It also makes ast::TokenKind more similar to parser::TokenType.

This requires a few new methods:

Despite these additions, it's a net reduction in lines of code. This is because e.g. token::OpenParen is so much shorter than token::OpenDelim(Delimiter::Parenthesis) that many multi-line forms reduce to single line forms. And many places where the number of lines doesn't change are still easier to read, just because the names are shorter, e.g.:

-   } else if self.token != token::CloseDelim(Delimiter::Brace) {
+   } else if self.token != token::CloseBrace {

bors added a commit to rust-lang-ci/rust that referenced this pull request

Apr 22, 2025

@bors

… r=petrochenkov

Remove token::{Open,Close}Delim

By replacing them with {Open,Close}{Param,Brace,Bracket,Invisible}.

PR rust-lang#137902 made ast::TokenKind more like lexer::TokenKind by replacing the compound BinOp{,Eq}(BinOpToken) variants with fieldless variants Plus, Minus, Star, etc. This commit does a similar thing with delimiters. It also makes ast::TokenKind more similar to parser::TokenType.

This requires a few new methods:

Despite these additions, it's a net reduction in lines of code. This is because e.g. token::OpenParen is so much shorter than token::OpenDelim(Delimiter::Parenthesis) that many multi-line forms reduce to single line forms. And many places where the number of lines doesn't change are still easier to read, just because the names are shorter, e.g.:

-   } else if self.token != token::CloseDelim(Delimiter::Brace) {
+   } else if self.token != token::CloseBrace {

r? @petrochenkov

andrew-otiv pushed a commit to andrew-otiv/rust that referenced this pull request

Apr 22, 2025

@nnethercote @andrew-otiv

By replacing them with {Open,Close}{Param,Brace,Bracket,Invisible}.

PR rust-lang#137902 made ast::TokenKind more like lexer::TokenKind by replacing the compound BinOp{,Eq}(BinOpToken) variants with fieldless variants Plus, Minus, Star, etc. This commit does a similar thing with delimiters. It also makes ast::TokenKind more similar to parser::TokenType.

This requires a few new methods:

Despite these additions, it's a net reduction in lines of code. This is because e.g. token::OpenParen is so much shorter than token::OpenDelim(Delimiter::Parenthesis) that many multi-line forms reduce to single line forms. And many places where the number of lines doesn't change are still easier to read, just because the names are shorter, e.g.:

-   } else if self.token != token::CloseDelim(Delimiter::Brace) {
+   } else if self.token != token::CloseBrace {

Labels

S-waiting-on-bors

Status: Waiting on bors to run and complete tests. Bors will change the label on completion.

T-compiler

Relevant to the compiler team, which will review and decide on the PR/issue.

T-rustdoc

Relevant to the rustdoc team, which will review and decide on the PR/issue.