proc_macro: Add an expand_expr method to TokenStream by mystor · Pull Request #87264 · rust-lang/rust (original) (raw)

We discussed this in the library-api meeting just now. We do like to see macro-expanding functionality on TokenStream. We do see the concerns with a general .expand() as already mentioned above, but would like to explore solutions that allow expanding to more than just literals in the near future. So we'd like to propose:

  1. Rename this to not be specific to just literals and return a TokenStream instead of a Literal; and
  2. Keep the implementation as restrictive as it is, failing if the result is not a single literal.

This way we sidestep the potential issues for now, but leave space for expanding the use cases of this function in the near future to include more expansions than only literals.

I'm not sure a general expand method would make sense, as macro expansion can be dependent on the context where it's occuring, and the TokenStream passed in needs to be parsed into an expression/type/whatever before expansion can start. This method could be generalized to expand_expr() rather than expand_literal(), but we'd probably also need expand_type(), expand_pat(), expand_items() etc. methods to cover the different types of macro expansions which are supported by rust. A general expand would probably need to either infer what is being expanded from how the macro being invoked is being expanded (though that feels sketchy to me, and I'd prefer not to do that), or would need to pick an expansion type like Expr and require wrapping other types in blocks or similar to parse items, types, or patterns.

If we do generalize to expand_expr(), I suppose we could also relax the requirements slightly to also allow true and false literal tokens now that they don't need to be valid Literals.

Could we provide a way to expand all macros recursively, and then if any macro expands to code that uses any unstable features internally, we expand that particular macro to an Opaque object whose output can be included in the output stream but not introspected or modified? That way, you can expand something like format_args! if you're only going to consume its output, but if you want to introspect the output of something (e.g. walk the tokens from an included file) it needs to be stable.

Having some way to represent an opaque sequence of tokens in the TokenStream would definitely be quite useful for this, but it would be quite a hassle to do that backwards compatibly at this point. The TokenTree enum is stable and only has the Ident, Literal, Punct, and Group options, so if we were to encode an opaque stream in, it would need to be done using one of those variants, as we can't add an Opaque variant.

The approach which seems most likely to me to not break existing macros would be to re-wrap the expanded AST as an unexpanded macro (e.g. opaque_expr!{}) with the specific expansion encoded into the span's context. This could theoretically be wrapped in a special Delimiter::None-delimited group with an as_opaque(&self) -> Option<Opaque> getter if we wanted to add special methods for operating on opaque expansions in the future.

Either way, this is probably something which should be handled in a separate issue or RFC.