clang: clang::Token Class Reference (original) (raw)

Token - This structure provides full information about a lexed token. More...

#include "[clang/Lex/Token.h](Token%5F8h%5Fsource.html)"

Token - This structure provides full information about a lexed token.

It is not intended to be space efficient, it is intended to return as much information as possible about each returned token. This is expected to be compressed into a smaller form if memory footprint is important.

The parser can create a special "annotation token" representing a stream of tokens that were parsed and semantically resolved, e.g.: "foo::MyClass" can be represented by a single typename annotation token that carries information about the SourceRange of the tokens and the type object.

Definition at line 36 of file Token.h.

◆ TokenFlags

Definition at line 74 of file Token.h.

◆ clearFlag()

◆ commaAfterElided()

◆ getAnnotationEndLoc()

◆ getAnnotationRange()

◆ getAnnotationValue()

◆ getEndLoc()

◆ getEofData()

◆ getFlag()

◆ getFlags()

Return the internal represtation of the flags.

This is only intended for low-level operations such as writing tokens to disk.

Definition at line 262 of file Token.h.

Referenced by clang::ASTWriter::AddToken().

◆ getIdentifierInfo()

Definition at line 187 of file Token.h.

References is(), isAnnotation(), isLiteral(), and isNot().

Referenced by clang::Sema::ActOnPragmaUnused(), AddDefinedMacroToHash(), clang::ASTWriter::AddToken(), clang::MacroArgs::ArgNeedsPreexpansion(), clang::TokenConcatenation::AvoidConcat(), clang::Preprocessor::CheckMacroName(), diagnoseAutoModuleImport(), dumpTokenInto(), EvaluateDefined(), EvaluateFeatureLikeBuiltinMacro(), EvaluateValue(), ExpectFeatureIdentifierInfo(), GetFirstChar(), getIdentifier(), getObjCKeywordID(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpelling(), clang::Preprocessor::HandleDirective(), clang::PragmaNamespace::HandlePragma(), isConfigurationPattern(), clang::format::isFunctionDeclarationName(), clang::MacroInfo::isIdenticalTo(), clang::format::AdditionalKeywords::isJavaScriptIdentifier(), isObjCAtKeyword(), isSameToken(), isSimpleTypeSpecifier(), isTrivialSingleTokenExpansion(), isWideStringLiteralFromMacro(), clang::format::AdditionalKeywords::isWordLike(), clang::TokenLexer::Lex(), clang::Preprocessor::LexEmbedParameters(), LexModuleNameComponent(), clang::Preprocessor::LexOnOffSwitch(), clang::MacroPPCallbacks::MacroDefined(), clang::detail::MacroExpansionRangeRecorder::MacroExpands(), clang::MacroPPCallbacks::MacroUndefined(), clang::format::mustBeJSIdent(), clang::TokenValue::operator==(), ParseAlignPragma(), parseDeclareSimdClauses(), clang::Parser::parseMapperModifier(), clang::Parser::parseMapTypeModifiers(), clang::Parser::ParseMSAsmIdentifier(), parseOpenMPReductionId(), clang::Parser::ParseOpenMPReservedLocator(), clang::Parser::ParseOpenMPVarList(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), PragmaLoopHintString(), PrintPreprocessedTokens(), clang::Parser::TryAnnotateTypeOrScopeToken(), clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(), and clang::Preprocessor::TypoCorrectToken().

◆ getKind()

Definition at line 94 of file Token.h.

Referenced by clang::Sema::ActOnCharacterConstant(), clang::Sema::ActOnNumericConstant(), clang::ASTWriter::AddToken(), atOrBeforeSeparation(), clang::TokenConcatenation::AvoidConcat(), clang::Sema::CheckConstraintExpression(), clang::Preprocessor::CheckMacroName(), clang::Lexer::ComputePreamble(), contains(), clang::Preprocessor::DumpToken(), EvaluateDirectiveSubExpr(), EvaluateFeatureLikeBuiltinMacro(), EvaluateHasIncludeCommon(), EvaluateValue(), getBeginningOfFileToken(), getEntityEndLoc(), getIdentifier(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), getSpellingSlow(), clang::Preprocessor::HandlePragmaModuleBuild(), isAnnotation(), isAnyIdentifier(), IsBuiltinTrait(), isConfigurationPattern(), clang::format::isConfiguredQualifier(), clang::MacroInfo::isIdenticalTo(), clang::format::AdditionalKeywords::isJavaScriptIdentifier(), isLiteral(), clang::format::FormatToken::isObjCAtKeyword(), clang::Preprocessor::IsPreviousCachedToken(), clang::format::isQualifier(), isRegularKeywordAttribute(), isSameToken(), isSimpleTypeSpecifier(), clang::Preprocessor::LexEmbedParameters(), clang::TokenValue::operator==(), clang::Parser::ParseConstraintLogicalAndExpression(), parseOpenMPReductionId(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), PrintPreprocessedTokens(), clang::ASTReader::ReadToken(), clang::CodeGen::CoverageMappingModuleGen::setUpCoverageCallbacks(), clang::format::ShouldBreakBeforeBrace(), clang::Parser::SkipMalformedDecl(), clang::Parser::SkipUntil(), clang::MacroArgs::StringifyArgument(), and clang::tokenIsLikeStringLiteral().

◆ getLastLoc()

◆ getLength()

Definition at line 135 of file Token.h.

References isAnnotation().

Referenced by clang::Sema::ActOnNumericConstant(), clang::ASTWriter::AddToken(), clang::TokenConcatenation::AvoidConcat(), clang::Preprocessor::DumpToken(), dumpTokenInto(), EmitAlphaCounter(), fixSeparateAttrArgAndNumber(), getBeginningOfFileToken(), getEndLoc(), GetFirstChar(), GetLineValue(), clang::StringLiteralParser::getOffsetOfStringByte(), getRawIdentifier(), clang::Preprocessor::getSpelling(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), getSpellingSlow(), clang::TokenLexer::Lex(), clang::Preprocessor::LexHeaderName(), clang::detail::MacroExpansionRangeRecorder::MacroExpands(), clang::Lexer::MeasureTokenLength(), NormalizeLine(), PrintPreprocessedTokens(), clang::format::IntegerLiteralSeparatorFixer::process(), clang::RewriteMacrosInInput(), clang::MacroArgs::StringifyArgument(), and UseNSOptionsMacro().

◆ getLiteralData()

getLiteralData - For a literal token (numeric constant, string, etc), this returns a pointer to the start of it in the text buffer if known, null otherwise.

Definition at line 225 of file Token.h.

References isLiteral().

Referenced by dumpTokenInto(), fixSeparateAttrArgAndNumber(), GetFirstChar(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), PrintPreprocessedTokens(), and UseNSOptionsMacro().

◆ getLocation()

Return a source location identifier for the specified offset in the current file.

Definition at line 132 of file Token.h.

References clang::SourceLocation::getFromRawEncoding(), and Loc.

Referenced by clang::Sema::ActOnCharacterConstant(), clang::Sema::ActOnNumericConstant(), clang::Sema::ActOnPragmaUnused(), clang::arcmt::trans::MigrationContext::addPropertyAttribute(), clang::ASTWriter::AddToken(), clang::TokenConcatenation::AvoidConcat(), clang::Preprocessor::CheckEndOfDirective(), clang::Preprocessor::CheckMacroName(), ComputeDATE_TIME(), clang::Lexer::ComputePreamble(), clang::Parser::ConsumeToken(), clang::Parser::Diag(), clang::Preprocessor::Diag(), diagnoseUnexpectedOperator(), clang::Preprocessor::DiscardUntilEndOfDirective(), clang::DoPrintPreprocessedInput(), clang::Preprocessor::DumpToken(), EvaluateDefined(), EvaluateDirectiveSubExpr(), EvaluateFeatureLikeBuiltinMacro(), EvaluateHasIncludeCommon(), EvaluateValue(), clang::Parser::ExitScope(), clang::Sema::ExpandFunctionLocalPredefinedMacros(), clang::BalancedDelimiterTracker::expectAndConsume(), ExpectFeatureIdentifierInfo(), findDirectives(), clang::arcmt::trans::findSemiAfterLocation(), FixDigraph(), clang::format::UnwrappedLineFormatter::format(), getAnnotationRange(), getBeginningOfFileToken(), getEndLoc(), GetFirstChar(), getLastLoc(), GetLineValue(), clang::StringLiteralParser::getOffsetOfStringByte(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), clang::Preprocessor::HandleDirective(), clang::Preprocessor::HandlePragmaDependency(), clang::Preprocessor::HandlePragmaHdrstop(), clang::Preprocessor::HandlePragmaIncludeAlias(), clang::Preprocessor::HandlePragmaMark(), clang::Preprocessor::HandlePragmaModuleBuild(), clang::Preprocessor::HandlePragmaPopMacro(), clang::Preprocessor::HandlePragmaSystemHeader(), clang::TokenLexer::Init(), clang::Preprocessor::IsPreviousCachedToken(), clang::TokenLexer::Lex(), clang::Preprocessor::LexEmbedParameters(), clang::Preprocessor::LexHeaderName(), LexModuleNameComponent(), LocPropertyAttribute(), clang::MacroPPCallbacks::MacroDefined(), clang::detail::MacroExpansionRangeRecorder::MacroExpands(), clang::MacroPPCallbacks::MacroUndefined(), clang::tooling::maybeExtendRange(), NormalizeLine(), ParseAlignPragma(), clang::Parser::ParseAssignmentExpression(), clang::Parser::ParseConditionalExpression(), parseDeclareSimdClauses(), ParseLoopHintValue(), clang::Parser::parseMapperModifier(), clang::Parser::parseMapTypeModifiers(), clang::Parser::ParseMSAsmIdentifier(), parseOpenMPAllocateClauseModifiers(), clang::Parser::ParseOpenMPParensExpr(), parseOpenMPReductionId(), parseOpenMPSimpleClause(), clang::Parser::ParseOpenMPVarList(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), clang::Preprocessor::parseSimpleIntegerLiteral(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), clang::format::IntegerLiteralSeparatorFixer::process(), ReadLineMarkerFlags(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), clang::CodeGen::CoverageMappingModuleGen::setUpCoverageCallbacks(), startsWithNewline(), clang::Parser::TryAnnotateTypeOrScopeToken(), clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(), clang::Parser::TryConsumeToken(), and updateConsecutiveMacroArgTokens().

◆ getName()

◆ getObjCKeywordID()

◆ getRawIdentifier()

◆ hasLeadingEmptyMacro()

◆ hasLeadingSpace()

◆ hasPtrData()

◆ hasUCN()

◆ hasUDSuffix()

◆ is()

is/isNot - Predicates to check if this token is a specific kind, as in "if (Tok.is(tok::l_brace)) {...}".

Definition at line 99 of file Token.h.

Referenced by clang::Sema::ActOnNameClassifiedAsNonType(), clang::arcmt::trans::MigrationContext::addPropertyAttribute(), clang::TokenConcatenation::AvoidConcat(), buildMSAsmString(), clang::Sema::CheckConstraintExpression(), clang::Preprocessor::CheckEndOfDirective(), clang::Preprocessor::CheckMacroName(), clang::Sema::ClassifyName(), clang::Parser::ConsumeAnyToken(), clang::Preprocessor::CreateString(), diagnoseUnexpectedOperator(), clang::DoPrintPreprocessedInput(), EvaluateDefined(), EvaluateHasIncludeCommon(), EvaluateValue(), findDirectives(), clang::Lexer::findPreviousToken(), getEofData(), getIdentifier(), getIdentifierInfo(), getRawIdentifier(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), clang::Preprocessor::HandleMicrosoftCommentPaste(), clang::Preprocessor::HandlePragmaHdrstop(), clang::Preprocessor::HandlePragmaIncludeAlias(), clang::Preprocessor::HandlePragmaModuleBuild(), clang::Preprocessor::HandlePragmaPoison(), clang::format::FormatToken::is(), clang::format::isAlignableBinaryOperator(), IsCommonTypo(), isMapModifier(), clang::TokenLexer::isNextTokenLParen(), isOneOf(), clang::TokenLexer::isParsingPreprocessorDirective(), isResultTypeOrTemplate(), isWideStringLiteralFromMacro(), clang::TokenLexer::Lex(), clang::Preprocessor::LexEmbedParameters(), clang::Preprocessor::LexHeaderName(), LexModuleNameComponent(), LexRawTokensFromMainFile(), LocPropertyAttribute(), clang::Parser::ConditionDeclarationOrInitStatementState::markNotExpression(), clang::tooling::maybeExtendRange(), clang::Parser::MightBeCXXScopeToken(), clang::IncrementalParser::Parse(), clang::Parser::ParseAssignmentExpression(), clang::Parser::ParseConditionalExpression(), clang::Parser::ParseConstraintLogicalAndExpression(), clang::Parser::ParseConstraintLogicalOrExpression(), parseDeclareSimdClauses(), ParseLoopHintValue(), parseMapType(), clang::Parser::parseMapTypeModifiers(), clang::Parser::ParseMSAsmIdentifier(), clang::Parser::ParseOpenACCDirectiveDecl(), clang::Parser::ParseOpenACCDirectiveStmt(), parseOpenMPAllocateClauseModifiers(), parseOpenMPReductionId(), clang::Parser::ParseOpenMPReservedLocator(), clang::Parser::ParseOpenMPVarList(), clang::Preprocessor::parseSimpleIntegerLiteral(), clang::Parser::ParseUnqualifiedId(), clang::format::CommaSeparatedList::precomputeFormattingInfos(), PrintPreprocessedTokens(), clang::format::IntegerLiteralSeparatorFixer::process(), ReadLineMarkerFlags(), clang::Lexer::ReadToEndOfLine(), clang::RewriteIncludesInInput(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), setEofData(), setRawIdentifierData(), clang::Preprocessor::SkipTokensWhileUsingPCH(), clang::Parser::SkipUntil(), clang::MacroArgs::StringifyArgument(), clang::format::switchesFormatting(), clang::Parser::TryAnnotateTypeOrScopeToken(), and clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec().

◆ isAnnotation()

Return true if this is any of tok::annot_* kind tokens.

Definition at line 121 of file Token.h.

References getKind(), and clang::tok::isAnnotation().

Referenced by clang::ASTWriter::AddToken(), clang::Preprocessor::AnnotateCachedTokens(), clang::TokenConcatenation::AvoidConcat(), clang::Parser::ConsumeAnyToken(), clang::Preprocessor::DumpToken(), dumpTokenInto(), ExpectFeatureIdentifierInfo(), getAnnotationEndLoc(), getAnnotationValue(), getEndLoc(), getIdentifierInfo(), getLastLoc(), getLength(), getObjCKeywordID(), isObjCAtKeyword(), clang::TokenLexer::Lex(), LexModuleNameComponent(), parseOpenMPDirectiveKind(), parseOpenMPSimpleClause(), clang::Parser::ParseOpenMPVarList(), PrintPreprocessedTokens(), clang::ASTReader::ReadToken(), clang::Preprocessor::ReplaceLastTokenWithAnnotation(), setAnnotationEndLoc(), setAnnotationValue(), setLength(), and clang::Parser::TryAnnotateTypeOrScopeToken().

◆ isAnyIdentifier()

◆ isAtStartOfLine()

isAtStartOfLine - Return true if this token is at the start of a line.

Definition at line 276 of file Token.h.

References getFlag(), and StartOfLine.

Referenced by buildMSAsmString(), clang::Lexer::ComputePreamble(), clang::Preprocessor::DumpToken(), clang::Preprocessor::HandlePragmaModuleBuild(), clang::TokenLexer::Init(), clang::MacroInfo::isIdenticalTo(), clang::Preprocessor::LexHeaderName(), NormalizeLine(), PrintPreprocessedTokens(), clang::RewriteMacrosInInput(), clang::Parser::SkipMalformedDecl(), and clang::MacroArgs::StringifyArgument().

◆ isEditorPlaceholder()

◆ isExpandDisabled()

◆ isLiteral()

Return true if this is a "literal", like a numeric constant, string, etc.

Definition at line 116 of file Token.h.

References getKind(), and clang::tok::isLiteral().

Referenced by clang::Preprocessor::CreateString(), dumpTokenInto(), GetFirstChar(), getIdentifierInfo(), getLiteralData(), clang::Lexer::getSpelling(), isWideStringLiteralFromMacro(), clang::format::mustBeJSIdentOrValue(), PrintPreprocessedTokens(), setLiteralData(), and UseNSOptionsMacro().

◆ isNot()

Definition at line 100 of file Token.h.

Referenced by clang::arcmt::trans::MigrationContext::addPropertyAttribute(), clang::MacroArgs::ArgNeedsPreexpansion(), clang::TokenConcatenation::AvoidConcat(), clang::Sema::ClassifyName(), clang::Preprocessor::DiscardUntilEndOfDirective(), dumpTokenInto(), EvaluateDefined(), EvaluateDirectiveSubExpr(), EvaluateFeatureLikeBuiltinMacro(), EvaluateHasIncludeCommon(), EvaluateValue(), clang::PrintDependencyDirectivesSourceMinimizerAction::ExecuteAction(), clang::DumpRawTokensAction::ExecuteAction(), clang::DumpTokensAction::ExecuteAction(), clang::PreprocessOnlyAction::ExecuteAction(), findDirectives(), clang::arcmt::trans::findSemiAfterLocation(), clang::MacroArgs::getArgLength(), getIdentifierInfo(), clang::format::getLastNonComment(), GetLineValue(), clang::Preprocessor::getSpelling(), clang::Preprocessor::HandleMicrosoftCommentPaste(), clang::Preprocessor::HandlePragmaDependency(), clang::Preprocessor::HandlePragmaHdrstop(), clang::Preprocessor::HandlePragmaIncludeAlias(), clang::Preprocessor::HandlePragmaModuleBuild(), clang::Preprocessor::HandlePragmaPoison(), clang::Preprocessor::LexEmbedParameters(), clang::Preprocessor::LexHeaderName(), LexModuleName(), clang::Preprocessor::LexOnOffSwitch(), LexRawTokensFromMainFile(), LocPropertyAttribute(), clang::Parser::ConditionDeclarationOrInitStatementState::markNotExpression(), clang::IncrementalParser::Parse(), ParseAlignPragma(), parseDeclareSimdClauses(), ParseLoopHintValue(), clang::Parser::parseMapperModifier(), clang::Parser::parseMapTypeModifiers(), clang::Parser::ParseMSAsmIdentifier(), parseOpenMPAllocateClauseModifiers(), parseOpenMPSimpleClause(), clang::Parser::ParseOpenMPVarList(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), parseStepSize(), clang::Parser::ParseUnqualifiedId(), clang::format::IntegerLiteralSeparatorFixer::process(), clang::RewriteIncludesInInput(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), clang::Parser::SkipUntil(), clang::MacroArgs::StringifyArgument(), clang::TokenRewriter::TokenRewriter(), and clang::Parser::TryConsumeToken().

◆ isObjCAtKeyword()

◆ isOneOf() [1/2]

Definition at line 101 of file Token.h.

References is().

Referenced by clang::TokenConcatenation::AvoidConcat(), clang::Sema::ClassifyName(), EvaluateFeatureLikeBuiltinMacro(), isConfigurationPattern(), isEndOfTemplateArgument(), isMapType(), clang::Preprocessor::LexEmbedParameters(), clang::Preprocessor::LexHeaderName(), clang::Preprocessor::LexTokensUntilEOF(), clang::Parser::ParseConstraintLogicalAndExpression(), clang::format::CommaSeparatedList::precomputeFormattingInfos(), and clang::Parser::SkipMalformedDecl().

◆ isOneOf() [2/2]

◆ isRegularKeywordAttribute()

◆ isSimpleTypeSpecifier()

◆ needsCleaning()

◆ setAnnotationEndLoc()

◆ setAnnotationRange()

◆ setAnnotationValue()

◆ setEofData()

◆ setFlag()

◆ setFlagValue()

◆ setIdentifierInfo()

◆ setKind()

Definition at line 95 of file Token.h.

Referenced by clang::Sema::ActOnIdExpression(), clang::TokenRewriter::AddTokenBefore(), clang::Preprocessor::EnterAnnotationToken(), EvaluateFeatureLikeBuiltinMacro(), clang::Sema::ExpandFunctionLocalPredefinedMacros(), findDirectives(), FixDigraph(), GenerateNewArgTokens(), clang::TokenLexer::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::Preprocessor::LexHeaderName(), ParseLoopHintValue(), clang::Parser::ParseMSAsmIdentifier(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), clang::Parser::Parser(), clang::Parser::ParseUnqualifiedId(), clang::ASTReader::ReadToken(), clang::format::ScopedMacroState::ScopedMacroState(), clang::MacroArgs::StringifyArgument(), clang::Parser::TryAnnotateTypeOrScopeToken(), clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(), and clang::VAOptExpansionContext::VAOptExpansionContext().

◆ setLength()

◆ setLiteralData()

◆ setLocation()

Definition at line 140 of file Token.h.

References clang::SourceLocation::getRawEncoding(), and Loc.

Referenced by clang::Sema::ActOnIdExpression(), clang::TokenRewriter::AddTokenBefore(), clang::Preprocessor::CreateString(), clang::Preprocessor::EnterAnnotationToken(), FixDigraph(), GenerateNewArgTokens(), clang::TokenLexer::Lex(), clang::Preprocessor::LexAfterModuleImport(), ParseLoopHintValue(), clang::ASTReader::ReadToken(), setAnnotationRange(), clang::Parser::TryAnnotateTypeOrScopeToken(), and clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec().

◆ setRawIdentifierData()

◆ startToken()

Reset all flags to cleared.

Definition at line 177 of file Token.h.

References clang::SourceLocation::getRawEncoding(), and Loc.

Referenced by clang::Sema::ActOnIdExpression(), clang::TokenRewriter::AddTokenBefore(), ComputeDATE_TIME(), clang::Preprocessor::EnterAnnotationToken(), clang::Sema::ExpandFunctionLocalPredefinedMacros(), GenerateNewArgTokens(), clang::format::JavaScriptImportSorter::JavaScriptImportSorter(), clang::TokenLexer::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::Preprocessor::LexHeaderName(), ParseLoopHintValue(), clang::Parser::ParseMSAsmIdentifier(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), clang::Parser::Parser(), clang::Lexer::ReadToEndOfLine(), clang::ASTReader::ReadToken(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), clang::format::ScopedMacroState::ScopedMacroState(), clang::MacroArgs::StringifyArgument(), and clang::VAOptExpansionContext::VAOptExpansionContext().

◆ stringifiedInMacro()

The documentation for this class was generated from the following files:

• include/clang/Lex/Token.h
• lib/Lex/Lexer.cpp