proc_macro: implement TokenTree, TokenKind, hygienic quote!, and other API by jseyfried · Pull Request #40939 · rust-lang/rust (original) (raw)

@eddyb

I'm a bit disappointed in how I fooled myself we could get rid of the "tree" model but flattening seems to not be in the cards.

Do you mean w.r.t. the internals? I believe we can still use the internal flattened representation you proposed with this API (imo, would be nice but not high priority).

@SimonSapin

Would impls for &TokenStream and &mut TokenStream also be useful?

impl IntoIterator for &TokenStream { type Item = TokenTree; ... } and/or
impl TokenStream { fn iter(&self) -> TokenIter { ... } } could be useful -- both would be equivalent to tokens.clone().into_iter().

impl IntoIterator for &TokenStream { type Item = &TokenTree; ... } isn't possible (yet, perhaps), and would make an internal "flattened" internal representation more complex / less efficient.

TokenStream uses shared, ref-counted memory, so I don't think impl IntoIterator for &mut TokenStream makes sense.

To me the word "symbol" evokes punctuation, excluding letters. Could this be called something else?

Yeah. Currently, Symbol is just an interned string, so Word/Ident don't seem appropriate. If we restrict the type to valid TokenKind::Words, I'd prefer Word for symmetry with TokenKind. Otherwise, perhaps Str/InternedStr? cc @nrc

So the = in += would be a separate token? &&, .., or :: would similarly be two tokens each?

Yeah, it's TokenKind::Op(char, OpKind), so a just single character per Op -- I'll mention that in the doc comment.

I’d also add [more methods for Literal]

Yeah, I agree that we should have Literal::{byte, byte_string, as_*}. This API isn't intended to be exhaustive, e.g. we also want API for emitting "first class" diagnostics (@sergio is working on this).

Why are there two quote macros, with one empty, and only the empty one exported?

The non-empty one is incomplete (by design) and only used to bootstrap the internal implementation of proc_macro::quote. The exported one is a placeholder for the internal implementation (à la lang items). After we support stability hygiene and stability-checking macro resolutions, we should be able to replace the internal implementation with a #[proc_macro].