parser: Keep current and previous tokens precisely by petrochenkov · Pull Request #69006 · rust-lang/rust (original) (raw)
Centril added a commit to Centril/rust that referenced this pull request
parser: token -> normalized_token, nonnormalized_token -> token
So, after rust-lang#69006, its follow-ups and an attempt to remove Parser::prev_span I came to the conclusion that the unnormalized token and its span is what you want in most cases, so it should be default.
Normalization only makes difference in few cases where we are checking against token::Ident or token::Lifetime specifically.
This PR uses normalized_token for those cases.
Using normalization explicitly means that people writing code should remember about NtIdent and NtLifetime in general. (That is alleviated by the fact that token.ident() and fn parse_ident_* are already written.)
Remembering about NtIdent, was, however, already the case, kind of, because the implicit normalization was performed only for the current/previous token, but not for things like look_ahead.
As a result, most of token classification methods in token.rs already take NtIdent into account (this PR fixes a few pre-existing minor mistakes though).
The next step is removing normalized(_prev)_token entirely and replacing it with token.ident() (mostly) and token.normalize() (occasionally).
I want to make it a separate PR for that and run it though perf.
normalized_token filled on every bump has both a potential to avoid repeated normalization, and to do unnecessary work in advance (it probably doesn't matter anyway, the normalization is very cheap).
r? @Centril