Token

Token.

class Token {

SourceFile _file;

int _line;

int _pos;

TokenType _type;

TokenType type [@property getter];

SourceFile filename [@property getter];

int line [@property getter];

int pos [@property getter];

dstring text [@property getter];

OpCode opCode [@property getter];

Keyword keyword [@property getter];

bool isDocumentationComment [@property getter];

bool isMultilineComment [@property getter];

bool isError [@property getter];

string errorMessage [@property getter];

int errorCode [@property getter];

TokenType invalidTokenType [@property getter];

void setPos(SourceFile file, int line, int pos);

void setFile(SourceFile file);

void setPos(int line, int pos);

}

Members

Functions

setFile void setFile(SourceFile file): set source file information for token
setPos void setPos(SourceFile file, int line, int pos): set start position for token (line is 1-based, pos is 0-based)
setPos void setPos(int line, int pos): set start position for token (line is 1-based, pos is 0-based)

Properties

errorCode int errorCode [@property getter]: returns error code if it's invalid token (can be returned in error tolerant mode of tokenizer)
errorMessage string errorMessage [@property getter]: returns error message if it's invalid token (can be returned in error tolerant mode of tokenizer)
filename SourceFile filename [@property getter]: returns file info for source
invalidTokenType TokenType invalidTokenType [@property getter]: returns type of token parsing of which has been failed - if it's invalid token (can be returned in error tolerant mode of tokenizer)
isDocumentationComment bool isDocumentationComment [@property getter]: returns true if this is documentation comment token
isError bool isError [@property getter]: returns true if it's invalid token (can be returned in error tolerant mode of tokenizer)
isMultilineComment bool isMultilineComment [@property getter]: returns true if this is multiline
keyword Keyword keyword [@property getter]: returns keyword ID - for keyword tokens
line int line [@property getter]: returns 1-based source line number of token start
opCode OpCode opCode [@property getter]: returns opcode ID - for opcode tokens
pos int pos [@property getter]: returns 1-based source line position of token start
text dstring text [@property getter]: returns token text
type TokenType type [@property getter]: returns token type

Meta

Source

See Implementation

ddc lexer tokenizer classes

functions

variables

identMap