proc_macro: Implementation of Extend (parameterized over Group, Ident, Punct, and Literal) for TokenStream · Issue #242 · rust-lang/libs-team (original) (raw)

Proposal

Problem statement

At the moment of writing, proc_macro::TokenStream implements parametrizations of core::iter::Extend over proc_macro::TokenStream and proc_macro::TokenTree, i.e. Extend<TokenStream> and Extend<TokenTree>. However, it is not implemented for parametrizations of Extend over

  1. proc_macro::Group
  2. proc_macro::Ident
  3. proc_macro::Punct
  4. proc_macro::Literal

which all can can be stored in respective variants of TokenTree.

I believe that this is a problem because it either

  1. causes unnecessary verbosity/boilerplate when one of Group, Ident, Punct, or Literal, is added to a TokenStream, as will be demonstrated further; or
  2. forces the developers to implement a local twin of Extend that they would implement for TokenStream, either manually or using a non-standard library.

Such inconveniences could push developers use quote, which results in higher compile times, even in situations where proc_macro with the proposed changes would suffice.

Also, it probably results in higher compile times in projects which extensively use procedural macros because these macros have to generate more Rust syntax.

Motivating examples or use cases

Before:

use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend(iter::once(TokenTree::Ident(Ident::new("new_ident", Span::call_site()))));

After:

use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend(iter::once(Ident::new("new_ident", Span::call_site())));

Once (or when) Extend::extend_one stabilizes, the difference will be even more noticeable.

Before:

use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend_one(TokenTree::Ident(Ident::new("new_ident", Span::call_site())));

After:

use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend_one(Ident::new("new_ident", Span::call_site()));

Solution sketch

impl Extend for TokenStream { fn extend<I: IntoIterator<Item = Group>>(&mut self, groups: I) { self.extend(groups.into_iter().map(TokenTree::Group)) } }

// And so on for the 3 remaining items, or we can use a macro_rules! for that.

Alternatives

Leave it as it is and delegate the implementation to a non-standard library via an extension trait that would be a twin of Extend. I dislike this approach because all these types belong to standard libraries and it would result in unnecessary complexity.