rustc_parser incorrectly parses groups with Delimiter::None · Issue #67062 · rust-lang/rust (original) (raw)

Such groups can currently be created by proc macros.

Reproduction, proc macro crate:

#[proc_macro] pub fn add_mul(_: TokenStream) -> TokenStream { // ⟪ 2 + 2 ⟫ * 2 vec![ TokenTree::from(Group::new(Delimiter::None, "2 + 2".parse().unwrap())), TokenTree::from(Punct::new('*', Spacing::Alone)), TokenTree::from(Literal::u8_unsuffixed(2)), ].into_iter().collect() }

User crate:

fn main() { let x = add_mul!(); assert_eq!(x, 8); // FAIL: the result is 6 != 8 }

This is not a huge issue right now because proc macros have no reasons to created Delimiter::None delimiters themselves (but they can still get them from input and return them if they return their input partially or fully, see the example below).

However, this will became a huge issue when interpolated tokens like $e: expr migrate from AST pieces to token streams, because in the token stream model they are represented exactly like groups with Delimiter::None delimiters (proc macros already see them in this way).