Remove TrailingToken. by nnethercote · Pull Request #127842 · rust-lang/rust (original) (raw)

Okay, maybe I'm a bit confused about the way in which the current machinery works, because it's not very consistent.
Maybe it depends on the parser code specifics whether capture_trailing should be set to true, not on node kinds.

If we parse like this

collect_tokens(parse_expr); eat_comma();

then we should pass capture_trailing = true to collect_tokens so it picks up the comma even if it's outside of collect_tokens.

If we parse like this

collect_tokens(parse_expr_and_comma);

then we should pass capture_trailing = false to collect_tokens because the comma is already inside collect_tokens and is collected.

Either way the comma will be collected, just through slightly different means, and will be included into the token stream for the expression.

I'm actually not sure anymore, will the commas appear in the token streams that my_proc_macro sees in

[#[my_proc_macro] a, #[my_proc_macro] b, #[my_proc_macro] c];

in the first two invocations?