ddc.lexer.tokenizer

Members

Classes

CommentToken

class CommentToken

comment token

IdentHolder

class IdentHolder

Global storage for identifier strings.

InvalidToken

class InvalidToken

Invalid token holder - for error tolerant parsing

Token

class Token

Token.

Tokenizer

class Tokenizer

WhiteSpaceToken

class WhiteSpaceToken

white space token

Functions

charToEntity

string charToEntity(dchar ch)

fings entity name for character, returns null if not found

entityToChar

dchar entityToChar(string name)

converts named entity to character, returns 0 if not found

isIdentMiddleChar

bool isIdentMiddleChar(dchar ch)

character can present in middle of identifier

isIdentStartChar

bool isIdentStartChar(dchar ch)

character can present at the beginning of identifier

isUniversalAlpha

bool isUniversalAlpha(dchar ch)

returns true if character is A..Z, a..z, _ or universal alpha

Variables

identMap

IdentHolder identMap;

Thread local storage for IDs.

• Lexer
• tokenizer