Tokens - The Rust Reference (original) (raw)

The Rust Reference

Tokens

Tokens are primitive productions in the grammar defined by regular (non-recursive) languages. Rust source input can be broken down into the following kinds of tokens:

Within this documentation’s grammar, “simple” tokens are given in string table production form, and appear in monospace font.

Literals

Literals are tokens used in literal expressions.

Examples

Characters and strings

Example # sets1 Characters Escapes
Character 'H' 0 All Unicode Quote & ASCII & Unicode
String "hello" 0 All Unicode Quote & ASCII & Unicode
Raw string r#"hello"# <256 All Unicode N/A
Byte b'H' 0 All ASCII Quote & Byte
Byte string b"hello" 0 All ASCII Quote & Byte
Raw byte string br#"hello"# <256 All ASCII N/A
C string c"hello" 0 All Unicode Quote & Byte & Unicode
Raw C string cr#"hello"# <256 All Unicode N/A

Note: Character and string literal tokens never include the sequence of U+000D (CR) immediately followed by U+000A (LF): this pair would have been previously transformed into a single U+000A (LF).

ASCII escapes

Name
\x41 7-bit character code (exactly 2 digits, up to 0x7F)
\n Newline
\r Carriage return
\t Tab
\\ Backslash
\0 Null

Byte escapes

Name
\x7F 8-bit character code (exactly 2 digits)
\n Newline
\r Carriage return
\t Tab
\\ Backslash
\0 Null

Unicode escapes

Name
\u{7FFF} 24-bit Unicode character code (up to 6 digits)

Quote escapes

Name
\' Single quote
\" Double quote

Numbers

Number literals2 Example Exponentiation
Decimal integer 98_222 N/A
Hex integer 0xff N/A
Octal integer 0o77 N/A
Binary integer 0b1111_0000 N/A
Floating-point 123.0E+77 Optional

Suffixes

A suffix is a sequence of characters following the primary part of a literal (without intervening whitespace), of the same form as a non-raw identifier or keyword.

Lexer
SUFFIX : IDENTIFIER_OR_KEYWORD
SUFFIX_NO_E : SUFFIX not beginning with e or E

Any kind of literal (string, integer, etc) with any suffix is valid as a token.

A literal token with any suffix can be passed to a macro without producing an error. The macro itself will decide how to interpret such a token and whether to produce an error or not. In particular, the literal fragment specifier for by-example macros matches literal tokens with arbitrary suffixes.

#![allow(unused)]
fn main() {
macro_rules! blackhole { ($tt:tt) => () }
macro_rules! blackhole_lit { ($l:literal) => () }

blackhole!("string"suffix); // OK
blackhole_lit!(1suffix); // OK
}

However, suffixes on literal tokens which are interpreted as literal expressions or patterns are restricted. Any suffixes are rejected on non-numeric literal tokens, and numeric literal tokens are accepted only with suffixes from the list below.

Integer Floating-point
u8, i8, u16, i16, u32, i32, u64, i64, u128, i128, usize, isize f32, f64

Character and string literals

Character literals

Lexer
CHAR_LITERAL :
' ( ~[' \ \n \r \t] | QUOTE_ESCAPE | ASCII_ESCAPE | UNICODE_ESCAPE ) ' SUFFIX?

QUOTE_ESCAPE :
\' | \"

ASCII_ESCAPE :
\x OCT_DIGIT HEX_DIGIT
| \n | \r | \t | \\ | \0

UNICODE_ESCAPE :
\u{ ( HEX_DIGIT _* )1..6 }

A character literal is a single Unicode character enclosed within twoU+0027 (single-quote) characters, with the exception of U+0027 itself, which must be escaped by a preceding U+005C character (\).

String literals

Lexer
STRING_LITERAL :
" (
~[" \ _IsolatedCR_]
| QUOTE_ESCAPE
| ASCII_ESCAPE
| UNICODE_ESCAPE
| STRING_CONTINUE
)* " SUFFIX?

STRING_CONTINUE :
\ followed by \n

A string literal is a sequence of any Unicode characters enclosed within twoU+0022 (double-quote) characters, with the exception of U+0022 itself, which must be escaped by a preceding U+005C character (\).

Line-breaks, represented by the character U+000A (LF), are allowed in string literals. When an unescaped U+005C character (\) occurs immediately before a line break, the line break does not appear in the string represented by the token. See String continuation escapes for details. The character U+000D (CR) may not appear in a string literal other than as part of such a string continuation escape.

Character escapes

Some additional escapes are available in either character or non-raw string literals. An escape starts with a U+005C (\) and continues with one of the following forms:

Raw string literals

Lexer
RAW_STRING_LITERAL :
r RAW_STRING_CONTENT SUFFIX?

RAW_STRING_CONTENT :
" ( ~ IsolatedCR )* (non-greedy) "
| # RAW_STRING_CONTENT #

Raw string literals do not process any escapes. They start with the characterU+0072 (r), followed by fewer than 256 of the character U+0023 (#) and aU+0022 (double-quote) character.

[lex.token.literal.str-raw.body]

The raw string body can contain any sequence of Unicode characters other than U+000D (CR). It is terminated only by another U+0022 (double-quote) character, followed by the same number of U+0023 (#) characters that preceded the opening U+0022 (double-quote) character.

[lex.token.literal.str-raw.content]

All Unicode characters contained in the raw string body represent themselves, the characters U+0022 (double-quote) (except when followed by at least as many U+0023 (#) characters as were used to start the raw string literal) orU+005C (\) do not have any special meaning.

Examples for string literals:

#![allow(unused)]
fn main() {
"foo"; r"foo";                     // foo
"\"foo\""; r#""foo""#;             // "foo"

"foo #\"# bar";
r##"foo #"# bar"##;                // foo #"# bar

"\x52"; "R"; r"R";                 // R
"\\x52"; r"\x52";                  // \x52
}

Byte and byte string literals

Byte literals

Lexer
BYTE_LITERAL :
b' ( ASCII_FOR_CHAR | BYTE_ESCAPE ) ' SUFFIX?

ASCII_FOR_CHAR :
any ASCII (i.e. 0x00 to 0x7F), except ', \, \n, \r or \t

BYTE_ESCAPE :
\x HEX_DIGIT HEX_DIGIT
| \n | \r | \t | \\ | \0 | \' | \"

A byte literal is a single ASCII character (in the U+0000 to U+007Frange) or a single escape preceded by the characters U+0062 (b) andU+0027 (single-quote), and followed by the character U+0027. If the characterU+0027 is present within the literal, it must be escaped by a precedingU+005C (\) character. It is equivalent to a u8 unsigned 8-bit integer_number literal_.

Byte string literals

Lexer
BYTE_STRING_LITERAL :
b" ( ASCII_FOR_STRING | BYTE_ESCAPE | STRING_CONTINUE )* " SUFFIX?

ASCII_FOR_STRING :
any ASCII (i.e 0x00 to 0x7F), except ", \ and IsolatedCR

A non-raw byte string literal is a sequence of ASCII characters and escapes, preceded by the characters U+0062 (b) and U+0022 (double-quote), and followed by the character U+0022. If the character U+0022 is present within the literal, it must be escaped by a preceding U+005C (\) character. Alternatively, a byte string literal can be a raw byte string literal, defined below.

Line-breaks, represented by the character U+000A (LF), are allowed in byte string literals. When an unescaped U+005C character (\) occurs immediately before a line break, the line break does not appear in the string represented by the token. See String continuation escapes for details. The character U+000D (CR) may not appear in a byte string literal other than as part of such a string continuation escape.

Some additional escapes are available in either byte or non-raw byte string literals. An escape starts with a U+005C (\) and continues with one of the following forms:

Raw byte string literals

Lexer
RAW_BYTE_STRING_LITERAL :
br RAW_BYTE_STRING_CONTENT SUFFIX?

RAW_BYTE_STRING_CONTENT :
" ASCII_FOR_RAW* (non-greedy) "
| # RAW_BYTE_STRING_CONTENT #

ASCII_FOR_RAW :
any ASCII (i.e. 0x00 to 0x7F) except IsolatedCR

Raw byte string literals do not process any escapes. They start with the character U+0062 (b), followed by U+0072 (r), followed by fewer than 256 of the character U+0023 (#), and a U+0022 (double-quote) character.

[lex.token.str-byte-raw.body]

The raw string body can contain any sequence of ASCII characters other than U+000D (CR). It is terminated only by another U+0022 (double-quote) character, followed by the same number of U+0023 (#) characters that preceded the opening U+0022 (double-quote) character. A raw byte string literal can not contain any non-ASCII byte.

[lex.token.literal.str-byte-raw.content]

All characters contained in the raw string body represent their ASCII encoding, the characters U+0022 (double-quote) (except when followed by at least as many U+0023 (#) characters as were used to start the raw string literal) orU+005C (\) do not have any special meaning.

Examples for byte string literals:

#![allow(unused)]
fn main() {
b"foo"; br"foo";                     // foo
b"\"foo\""; br#""foo""#;             // "foo"

b"foo #\"# bar";
br##"foo #"# bar"##;                 // foo #"# bar

b"\x52"; b"R"; br"R";                // R
b"\\x52"; br"\x52";                  // \x52
}

C string and raw C string literals

C string literals

Lexer
C_STRING_LITERAL :
c" (
~[" \ IsolatedCR _NUL_]
| BYTE_ESCAPE except \0 or \x00
| UNICODE_ESCAPE except \u{0}, \u{00}, …, \u{000000}
| STRING_CONTINUE
)* " SUFFIX?

A C string literal is a sequence of Unicode characters and escapes, preceded by the characters U+0063 (c) and U+0022 (double-quote), and followed by the character U+0022. If the character U+0022 is present within the literal, it must be escaped by a preceding U+005C (\) character. Alternatively, a C string literal can be a raw C string literal, defined below.

C strings are implicitly terminated by byte 0x00, so the C string literalc"" is equivalent to manually constructing a &CStr from the byte string literal b"\x00". Other than the implicit terminator, byte 0x00 is not permitted within a C string.

Line-breaks, represented by the character U+000A (LF), are allowed in C string literals. When an unescaped U+005C character (\) occurs immediately before a line break, the line break does not appear in the string represented by the token. See String continuation escapes for details. The character U+000D (CR) may not appear in a C string literal other than as part of such a string continuation escape.

Some additional escapes are available in non-raw C string literals. An escape starts with a U+005C (\) and continues with one of the following forms:

A C string represents bytes with no defined encoding, but a C string literal may contain Unicode characters above U+007F. Such characters will be replaced with the bytes of that character’s UTF-8 representation.

The following C string literals are equivalent:

#![allow(unused)]
fn main() {
c"æ";        // LATIN SMALL LETTER AE (U+00E6)
c"\u{00E6}";
c"\xC3\xA6";
}

Edition differences: C string literals are accepted in the 2021 edition or later. In earlier additions the token c"" is lexed as c "".

Raw C string literals

Lexer
RAW_C_STRING_LITERAL :
cr RAW_C_STRING_CONTENT SUFFIX?

RAW_C_STRING_CONTENT :
" ( ~ IsolatedCR NUL )* (non-greedy) "
| # RAW_C_STRING_CONTENT #

Raw C string literals do not process any escapes. They start with the character U+0063 (c), followed by U+0072 (r), followed by fewer than 256 of the character U+0023 (#), and a U+0022 (double-quote) character.

[lex.token.str-c-raw.body]

The raw C string body can contain any sequence of Unicode characters other than U+0000 (NUL) and U+000D (CR). It is terminated only by another U+0022 (double-quote) character, followed by the same number of U+0023 (#) characters that preceded the opening U+0022 (double-quote) character.

[lex.token.str-c-raw.content]

All characters contained in the raw C string body represent themselves in UTF-8 encoding. The characters U+0022 (double-quote) (except when followed by at least as many U+0023 (#) characters as were used to start the raw C string literal) or U+005C (\) do not have any special meaning.

Edition differences: Raw C string literals are accepted in the 2021 edition or later. In earlier additions the token cr"" is lexed as cr "", and cr#""# is lexed as cr #""# (which is non-grammatical).

Examples for C string and raw C string literals

#![allow(unused)]
fn main() {
c"foo"; cr"foo";                     // foo
c"\"foo\""; cr#""foo""#;             // "foo"

c"foo #\"# bar";
cr##"foo #"# bar"##;                 // foo #"# bar

c"\x52"; c"R"; cr"R";                // R
c"\\x52"; cr"\x52";                  // \x52
}

Number literals

A number literal is either an integer literal or a floating-point literal. The grammar for recognizing the two kinds of literals is mixed.

Integer literals

Lexer
INTEGER_LITERAL :
( DEC_LITERAL | BIN_LITERAL | OCT_LITERAL | HEX_LITERAL ) SUFFIX_NO_E?

DEC_LITERAL :
DEC_DIGIT (DEC_DIGIT|_)*

BIN_LITERAL :
0b (BIN_DIGIT|_)* BIN_DIGIT (BIN_DIGIT|_)*

OCT_LITERAL :
0o (OCT_DIGIT|_)* OCT_DIGIT (OCT_DIGIT|_)*

HEX_LITERAL :
0x (HEX_DIGIT|_)* HEX_DIGIT (HEX_DIGIT|_)*

BIN_DIGIT : [0-1]

OCT_DIGIT : [0-7]

DEC_DIGIT : [0-9]

HEX_DIGIT : [0-9 a-f A-F]

An integer literal has one of four forms:

Like any literal, an integer literal may be followed (immediately, without any spaces) by a suffix as described above. The suffix may not begin with e or E, as that would be interpreted as the exponent of a floating-point literal. See Integer literal expressions for the effect of these suffixes.

Examples of integer literals which are accepted as literal expressions:

#![allow(unused)]
fn main() {
#![allow(overflowing_literals)]
123;
123i32;
123u32;
123_u32;

0xff;
0xff_u8;
0x01_f32; // integer 7986, not floating-point 1.0
0x01_e3;  // integer 483, not floating-point 1000.0

0o70;
0o70_i16;

0b1111_1111_1001_0000;
0b1111_1111_1001_0000i64;
0b________1;

0usize;

// These are too big for their type, but are accepted as literal expressions.
128_i8;
256_u8;

// This is an integer literal, accepted as a floating-point literal expression.
5f32;
}

Note that -1i8, for example, is analyzed as two tokens: - followed by 1i8.

Examples of integer literals which are not accepted as literal expressions:

#![allow(unused)]
fn main() {
#[cfg(FALSE)] {
0invalidSuffix;
123AFB43;
0b010a;
0xAB_CD_EF_GH;
0b1111_f32;
}
}

Tuple index

Lexer
TUPLE_INDEX:
INTEGER_LITERAL

A tuple index is used to refer to the fields of tuples, tuple structs, andtuple variants.

Tuple indices are compared with the literal token directly. Tuple indices start with 0 and each successive index increments the value by 1 as a decimal value. Thus, only decimal values will match, and the value must not have any extra 0 prefix characters.

#![allow(unused)]
fn main() {
let example = ("dog", "cat", "horse");
let dog = example.0;
let cat = example.1;
// The following examples are invalid.
let cat = example.01;  // ERROR no field named `01`
let horse = example.0b10;  // ERROR no field named `0b10`
}

Note: Tuple indices may include certain suffixes, but this is not intended to be valid, and may be removed in a future version. See https://github.com/rust-lang/rust/issues/60210 for more information.

Floating-point literals

Lexer
FLOAT_LITERAL :
DEC_LITERAL . (not immediately followed by ., `` or an XID_Start character)
| DEC_LITERAL . DEC_LITERAL SUFFIX_NO_E?
| DEC_LITERAL (. DEC_LITERAL)? FLOAT_EXPONENT SUFFIX?

FLOAT_EXPONENT :
(e|E) (+|-)?(DEC_DIGIT|_)* DEC_DIGIT (DEC_DIGIT|_)*

A floating-point literal has one of two forms:

Like integer literals, a floating-point literal may be followed by a suffix, so long as the pre-suffix part does not end with U+002E (.). The suffix may not begin with e or E if the literal does not include an exponent. See Floating-point literal expressions for the effect of these suffixes.

Examples of floating-point literals which are accepted as literal expressions:

#![allow(unused)]
fn main() {
123.0f64;
0.1f64;
0.1f32;
12E+99_f64;
let x: f64 = 2.;
}

This last example is different because it is not possible to use the suffix syntax with a floating point literal end.token.ing in a period. 2.f64 would attempt to call a method named f64 on 2.

Note that -1.0, for example, is analyzed as two tokens: - followed by 1.0.

Examples of floating-point literals which are not accepted as literal expressions:

#![allow(unused)]
fn main() {
#[cfg(FALSE)] {
2.0f80;
2e5f80;
2e5e6;
2.0e5e6;
1.3e10u64;
}
}

Reserved forms similar to number literals

Lexer
RESERVED_NUMBER :
BIN_LITERAL [2-9​]
| OCT_LITERAL [8-9​]
| ( BIN_LITERAL | OCT_LITERAL | HEX_LITERAL ) .
(not immediately followed by ., `` or an XID_Start character)
| ( BIN_LITERAL | OCT_LITERAL ) (e|E)
| 0b _* end of input or not BIN_DIGIT
| 0o _* end of input or not OCT_DIGIT
| 0x _* end of input or not HEX_DIGIT
| DEC_LITERAL ( . DEC_LITERAL)? (e|E) (+|-)? end of input or not DEC_DIGIT

The following lexical forms similar to number literals are reserved forms. Due to the possible ambiguity these raise, they are rejected by the tokenizer instead of being interpreted as separate tokens.

Examples of reserved forms:

#![allow(unused)]
fn main() {
0b0102;  // this is not `0b010` followed by `2`
0o1279;  // this is not `0o127` followed by `9`
0x80.0;  // this is not `0x80` followed by `.` and `0`
0b101e;  // this is not a suffixed literal, or `0b101` followed by `e`
0b;      // this is not an integer literal, or `0` followed by `b`
0b_;     // this is not an integer literal, or `0` followed by `b_`
2e;      // this is not a floating-point literal, or `2` followed by `e`
2.0e;    // this is not a floating-point literal, or `2.0` followed by `e`
2em;     // this is not a suffixed literal, or `2` followed by `em`
2.0em;   // this is not a suffixed literal, or `2.0` followed by `em`
}

Lifetimes and loop labels

Lexer
LIFETIME_TOKEN :
' IDENTIFIER_OR_KEYWORD (not immediately followed by ')
| '_ (not immediately followed by ')
| RAW_LIFETIME

LIFETIME_OR_LABEL :
' NON_KEYWORD_IDENTIFIER (not immediately followed by ')
| RAW_LIFETIME

RAW_LIFETIME :
'r# IDENTIFIER_OR_KEYWORD Except crate, self, super, Self (not immediately followed by ')

RESERVED_RAW_LIFETIME : 'r#_ (not immediately followed by ')

Lifetime parameters and loop labels use LIFETIME_OR_LABEL tokens. Any LIFETIME_TOKEN will be accepted by the lexer, and for example, can be used in macros.

A raw lifetime is like a normal lifetime, but its identifier is prefixed by r#. (Note that the r# prefix is not included as part of the actual lifetime.)

Unlike a normal lifetime, a raw lifetime may be any strict or reserved keyword except the ones listed above for RAW_LIFETIME.

It is an error to use the RESERVED_RAW_LIFETIME token 'r#_ in order to avoid confusion with the placeholder lifetime.

Edition differences: Raw lifetimes are accepted in the 2021 edition or later. In earlier additions the token 'r#lt is lexed as 'r # lt.

Punctuation

Punctuation symbol tokens are listed here for completeness. Their individual usages and meanings are defined in the linked pages.

Symbol Name Usage
+ Plus Addition, Trait Bounds, Macro Kleene Matcher
- Minus Subtraction, Negation
* Star Multiplication, Dereference, Raw Pointers, Macro Kleene Matcher, Use wildcards
/ Slash Division
% Percent Remainder
^ Caret Bitwise and Logical XOR
! Not Bitwise and Logical NOT, Macro Calls, Inner Attributes, Never Type, Negative impls
& And Bitwise and Logical AND, Borrow, References, Reference patterns
| Or Bitwise and Logical OR, Closures, Patterns in match, if let, and while let
&& AndAnd Lazy AND, Borrow, References, Reference patterns
| OrOr
<< Shl Shift Left, Nested Generics
>> Shr Shift Right, Nested Generics
+= PlusEq Addition assignment
-= MinusEq Subtraction assignment
*= StarEq Multiplication assignment
/= SlashEq Division assignment
%= PercentEq Remainder assignment
^= CaretEq Bitwise XOR assignment
&= AndEq Bitwise And assignment
|= OrEq Bitwise Or assignment
<<= ShlEq Shift Left assignment
>>= ShrEq Shift Right assignment, Nested Generics
= Eq Assignment, Attributes, Various type definitions
== EqEq Equal
!= Ne Not Equal
> Gt Greater than, Generics, Paths
< Lt Less than, Generics, Paths
>= Ge Greater than or equal to, Generics
<= Le Less than or equal to
@ At Subpattern binding
_ Underscore Wildcard patterns, Inferred types, Unnamed items in constants, extern crates, use declarations, and destructuring assignment
. Dot Field access, Tuple index
.. DotDot Range, Struct expressions, Patterns, Range Patterns
... DotDotDot Variadic functions, Range patterns
..= DotDotEq Inclusive Range, Range patterns
, Comma Various separators
; Semi Terminator for various items and statements, Array types
: Colon Various separators
:: PathSep Path separator
-> RArrow Function return type, Closure return type, Function pointer type
=> FatArrow Match arms, Macros
<- LArrow The left arrow symbol has been unused since before Rust 1.0, but it is still treated as a single token
# Pound Attributes
$ Dollar Macros
? Question Question mark operator, Questionably sized, Macro Kleene Matcher
~ Tilde The tilde operator has been unused since before Rust 1.0, but its token may still be used

Delimiters

Bracket punctuation is used in various parts of the grammar. An open bracket must always be paired with a close bracket. Brackets and the tokens within them are referred to as “token trees” in macros. The three types of brackets are:

Bracket Type
{ } Curly braces
[ ] Square brackets
( ) Parentheses

Reserved prefixes

Lexer 2021+
RESERVED_TOKEN_DOUBLE_QUOTE : ( IDENTIFIER_OR_KEYWORD Except b or c or r or br or cr | _ ) "
RESERVED_TOKEN_SINGLE_QUOTE : ( IDENTIFIER_OR_KEYWORD Except b | _ ) '
RESERVED_TOKEN_POUND : ( IDENTIFIER_OR_KEYWORD Except r or br or cr | _ ) #
RESERVED_TOKEN_LIFETIME : ' (IDENTIFIER_OR_KEYWORD Except r | _) #

Some lexical forms known as reserved prefixes are reserved for future use.

Source input which would otherwise be lexically interpreted as a non-raw identifier (or a keyword or _) which is immediately followed by a #, ', or " character (without intervening whitespace) is identified as a reserved prefix.

Note that raw identifiers, raw string literals, and raw byte string literals may contain a # character but are not interpreted as containing a reserved prefix.

Similarly the r, b, br, c, and cr prefixes used in raw string literals, byte literals, byte string literals, raw byte string literals, C string literals, and raw C string literals are not interpreted as reserved prefixes.

Source input which would otherwise be lexically interpreted as a non-raw lifetime (or a keyword or _) which is immediately followed by a # character (without intervening whitespace) is identified as a reserved lifetime prefix.

Edition differences: Starting with the 2021 edition, reserved prefixes are reported as an error by the lexer (in particular, they cannot be passed to macros).

Before the 2021 edition, reserved prefixes are accepted by the lexer and interpreted as multiple tokens (for example, one token for the identifier or keyword, followed by a # token).

Examples accepted in all editions:

#![allow(unused)]
fn main() {
macro_rules! lexes {($($_:tt)*) => {}}
lexes!{a #foo}
lexes!{continue 'foo}
lexes!{match "..." {}}
lexes!{r#let#foo}         // three tokens: r#let # foo
lexes!{'prefix #lt}
}

Examples accepted before the 2021 edition but rejected later:

#![allow(unused)]
fn main() {
macro_rules! lexes {($($_:tt)*) => {}}
lexes!{a#foo}
lexes!{continue'foo}
lexes!{match"..." {}}
lexes!{'prefix#lt}
}

Reserved guards

Lexer 2024+
RESERVED_GUARDED_STRING_LITERAL : #+ STRING_LITERAL
RESERVED_POUNDS : #2..

The reserved guards are syntax reserved for future use, and will generate a compile error if used.

The reserved guarded string literal is a token of one or more U+0023 (#) immediately followed by a STRING_LITERAL.

The reserved pounds is a token of two or more U+0023 (#).

Edition differences: Before the 2024 edition, reserved guards are accepted by the lexer and interpreted as multiple tokens. For example, the #"foo"# form is interpreted as three tokens. ## is interpreted as two tokens.