Lexer in rustc_parse::lexer - Rust (original) (raw)

struct Lexer<'psess, 'src> {
    psess: &'psess ParseSess,
    start_pos: BytePos,
    pos: BytePos,
    src: &'src str,
    cursor: Cursor<'src>,
    override_span: Option<Span>,
    nbsp_is_whitespace: bool,
    last_lifetime: Option<Span>,
    token: Token,
    diag_info: TokenTreeDiagInfo,
}

Initial position, read-only.

The absolute offset within the source_map of the current character.

Source text to tokenize.

Cursor for getting lexer tokens.

When a “unknown start of token: \u{a0}” has already been emitted earlier in this file, it’s safe to treat further occurrences of the non-breaking space character as whitespace.

Track the Span for the leading ' of the last lifetime. Used for diagnostics to detect possible typo where " was meant.

The current token.

Returns the next token, paired with a bool indicating if the token was preceded by whitespace.

Source

Detect usages of Unicode codepoints changing the direction of the text on screen and loudly complain about it.

Source

Slice of the source text from start up to but excluding self.pos, meaning the slice does not include the character self.ch.

Source

As symbol_from, with an explicit endpoint.

Source

Slice of the source text spanning from start up to but excluding end.

Source

Slice of the source text spanning from start until the end

Source

Detect guarded string literal syntax

RFC 3593 reserved this syntax for future use. As of Rust 2024, using this syntax produces an error. In earlier editions, however, it only results in an (allowed by default) lint, and is treated as separate tokens.

Source

Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...) attributes. Please see the Rust Reference's “Type Layout” chapter for details on type layout guarantees.

Size: 232 bytes