Tokenizer

allTokens Token[] allTokens(): tokenize all
detectKeyword Keyword detectKeyword(dchar ch): Undocumented in source. Be warned that the author may not have intended to support it.
detectOp OpCode detectOp(dchar ch): Undocumented in source. Be warned that the author may not have intended to support it.
emitEof Token emitEof(): Undocumented in source. Be warned that the author may not have intended to support it.
formatBuildDate dstring formatBuildDate(): Undocumented in source. Be warned that the author may not have intended to support it.
formatBuildTime dstring formatBuildTime(): Undocumented in source. Be warned that the author may not have intended to support it.
formatBuildTimestamp dstring formatBuildTimestamp(): Undocumented in source. Be warned that the author may not have intended to support it.
initialize void initialize(SourceLines lineStream, int pos): Undocumented in source. Be warned that the author may not have intended to support it.
makeSpecialTokenString Token makeSpecialTokenString(dstring str, int pos): Undocumented in source. Be warned that the author may not have intended to support it.
nextChar dchar nextChar(): Undocumented in source. Be warned that the author may not have intended to support it.
nextLine bool nextLine(): Undocumented in source. Be warned that the author may not have intended to support it.
nextToken Token nextToken(): Undocumented in source. Be warned that the author may not have intended to support it.
parserError Token parserError(string msg, Token incompleteToken): Either return InvalidToken or throw parser exception depending on current errorTolerant flag
parserError Token parserError(string msg, int startLine, int startPos, TokenType failedTokenType): Either return InvalidToken or throw parser exception depending on current errorTolerant flag
peekChar dchar peekChar(): Undocumented in source. Be warned that the author may not have intended to support it.
processBinaryNumber Token processBinaryNumber(): Undocumented in source. Be warned that the author may not have intended to support it.
processCharacterLiteral Token processCharacterLiteral(): Undocumented in source. Be warned that the author may not have intended to support it.
processDecFloatExponent Token processDecFloatExponent(real value): Undocumented in source. Be warned that the author may not have intended to support it.
processDecFloatSecondPart Token processDecFloatSecondPart(ulong firstPart): Undocumented in source. Be warned that the author may not have intended to support it.
processDecFloatSuffix Token processDecFloatSuffix(real value): Undocumented in source. Be warned that the author may not have intended to support it.
processDecNumber Token processDecNumber(dchar c): Undocumented in source. Be warned that the author may not have intended to support it.
processDelimitedString Token processDelimitedString(): Undocumented in source. Be warned that the author may not have intended to support it.
processDoubleQuotedOrWysiwygString Token processDoubleQuotedOrWysiwygString(dchar delimiter): Undocumented in source. Be warned that the author may not have intended to support it.
processHexNumber Token processHexNumber(): Undocumented in source. Be warned that the author may not have intended to support it.
processHexString Token processHexString(): Undocumented in source. Be warned that the author may not have intended to support it.
processIdent Token processIdent(dchar firstChar): Undocumented in source. Be warned that the author may not have intended to support it.
processIntegerSuffix Token processIntegerSuffix(): Undocumented in source. Be warned that the author may not have intended to support it.
processMultilineComment Token processMultilineComment(): Undocumented in source. Be warned that the author may not have intended to support it.
processNestedComment Token processNestedComment(): Undocumented in source. Be warned that the author may not have intended to support it.
processOctNumber Token processOctNumber(): Undocumented in source. Be warned that the author may not have intended to support it.
processOneLineComment Token processOneLineComment(): Undocumented in source. Be warned that the author may not have intended to support it.
processOneLineSharpComment Token processOneLineSharpComment(): Undocumented in source. Be warned that the author may not have intended to support it.
processSpecialToken Token processSpecialToken(Keyword keyword, int pos): Undocumented in source. Be warned that the author may not have intended to support it.
processWhiteSpace Token processWhiteSpace(dchar firstChar): Undocumented in source. Be warned that the author may not have intended to support it.
processWysiwygString Token processWysiwygString(dchar ch): Undocumented in source. Be warned that the author may not have intended to support it.

Properties

enableCommentText bool enableCommentText [@property setter]: when false, does not put comment text into comment token - for less allocations
enableCommentText bool enableCommentText [@property getter]: when false, does not put comment text into comment token - for less allocations
errorTolerant bool errorTolerant [@property setter]: when true, returns BadToken instead of throwing exception
errorTolerant bool errorTolerant [@property getter]: when true, returns BadToken instead of throwing exception

Static variables

VENDOR dstring VENDOR;: Undocumented in source.
VERSION dstring VERSION;: Undocumented in source.

Variables

_commentAppender StringAppender _commentAppender;: Undocumented in source.
_enableCommentText bool _enableCommentText;: Undocumented in source.
_errorTolerant bool _errorTolerant;: Undocumented in source.
_identAppender StringAppender _identAppender;: Undocumented in source.
_len int _len;: Undocumented in source.
_line int _line;: Undocumented in source.
_lineStream SourceLines _lineStream;: Undocumented in source.
_lineText dchar[] _lineText;: Undocumented in source.
_pos int _pos;: Undocumented in source.
_prevLineLength int _prevLineLength;: Undocumented in source.
_sharedCharacterLiteralToken CharacterLiteralToken _sharedCharacterLiteralToken;: Undocumented in source.
_sharedCommentToken CommentToken _sharedCommentToken;: Undocumented in source.
_sharedIdentToken IdentToken _sharedIdentToken;: Undocumented in source.
_sharedIntegerToken IntegerLiteralToken _sharedIntegerToken;: Undocumented in source.
_sharedInvalidToken InvalidToken _sharedInvalidToken;: Undocumented in source.
_sharedKeywordToken KeywordToken _sharedKeywordToken;: Undocumented in source.
_sharedOpToken OpToken _sharedOpToken;: Undocumented in source.
_sharedRealToken RealLiteralToken _sharedRealToken;: Undocumented in source.
_sharedStringLiteralToken StringLiteralToken _sharedStringLiteralToken;: Undocumented in source.
_sharedWhiteSpaceToken WhiteSpaceToken _sharedWhiteSpaceToken;: Undocumented in source.
_startLine int _startLine;: Undocumented in source.
_startPos int _startPos;: Undocumented in source.
_state uint _state;: Undocumented in source.
_stringLiteralAppender StringAppender _stringLiteralAppender;: Undocumented in source.
buildTime SysTime buildTime;: Undocumented in source.