ddc.lexer.tokenizer

Undocumented in source.

Members

tokenizer_ident_name_t alias tokenizer_ident_name_t = dstring: Undocumented in source.
tokenizer_ident_t alias tokenizer_ident_t = uint: Undocumented in source.

CharacterLiteralToken class CharacterLiteralToken: Undocumented in source.
CommentToken class CommentToken: comment token
EofToken class EofToken: Undocumented in source.
IdentHolder class IdentHolder: Global storage for identifier strings.
IdentToken class IdentToken: Undocumented in source.
IntegerLiteralToken class IntegerLiteralToken: Undocumented in source.
InvalidToken class InvalidToken: Invalid token holder - for error tolerant parsing
KeywordToken class KeywordToken: Undocumented in source.
OpToken class OpToken: Undocumented in source.
RealLiteralToken class RealLiteralToken: Undocumented in source.
StringLiteralToken class StringLiteralToken: Undocumented in source.
Token class Token: Token.
Tokenizer class Tokenizer: Undocumented in source.
WhiteSpaceToken class WhiteSpaceToken: white space token

charToEntity string charToEntity(dchar ch): fings entity name for character, returns null if not found
entityToChar dchar entityToChar(string name): converts named entity to character, returns 0 if not found
findKeyword Keyword findKeyword(Keyword start, Keyword end, dchar* name, int len, int pos): Undocumented in source. Be warned that the author may not have intended to support it.
getKeywordNameD dstring getKeywordNameD(Keyword keyword): Undocumented in source. Be warned that the author may not have intended to support it.
getOpNameD dstring getOpNameD(OpCode op): Undocumented in source. Be warned that the author may not have intended to support it.
isIdentMiddleChar bool isIdentMiddleChar(dchar ch): character can present in middle of identifier
isIdentStartChar bool isIdentStartChar(dchar ch): character can present at the beginning of identifier
isUniversalAlpha bool isUniversalAlpha(dchar ch): returns true if character is A..Z, a..z, _ or universal alpha
isUniversalAlphaSlow bool isUniversalAlphaSlow(dchar c): Undocumented in source. Be warned that the author may not have intended to support it.
r bool r(dchar ch, wchar v): Undocumented in source. Be warned that the author may not have intended to support it.
r bool r(dchar ch, wchar v1, wchar v2): Undocumented in source. Be warned that the author may not have intended to support it.

ENABLE_DUMP_UNIVERSAL_ALPHA_TABLE bool ENABLE_DUMP_UNIVERSAL_ALPHA_TABLE;: Undocumented in source.
KEYWORD_STRINGS dstring[] KEYWORD_STRINGS;: Undocumented in source.
OP_CODE_STRINGS dstring[] OP_CODE_STRINGS;: Undocumented in source.
UNIVERSAL_ALPHA_FLAGS uint[1728] UNIVERSAL_ALPHA_FLAGS;: Undocumented in source.
identMap IdentHolder identMap;: Thread local storage for IDs.