Public Member Functions | |
| Token (int length, byte id) | |
| String | toString () |
Public Attributes | |
| int | length |
| byte | id |
| Token | next |
Static Public Attributes | |
| static final byte | NULL = 0 |
| static final byte | COMMENT1 = 1 |
| static final byte | COMMENT2 = 2 |
| static final byte | LITERAL1 = 3 |
| static final byte | LITERAL2 = 4 |
| static final byte | LABEL = 5 |
| static final byte | KEYWORD1 = 6 |
| static final byte | KEYWORD2 = 7 |
| static final byte | KEYWORD3 = 8 |
| static final byte | OPERATOR = 9 |
| static final byte | INVALID = 10 |
| static final byte | ID_COUNT = 11 |
| static final byte | INTERNAL_FIRST = 100 |
| static final byte | INTERNAL_LAST = 126 |
| static final byte | END = 127 |
|
||||||||||||
|
Creates a new token.
|
|
|
Returns a string representation of this token. |
|
|
Comment 1 token id. This can be used to mark a comment. |
|
|
Comment 2 token id. This can be used to mark a comment. |
|
|
The token type, that along with a length of 0 marks the end of the token list. |
|
|
The id of this token. |
|
|
The total number of defined token ids. |
|
|
The first id that can be used for internal state in a token marker. |
|
|
The last id that can be used for internal state in a token marker. |
|
|
Invalid token id. This can be used to mark invalid or incomplete tokens, so the user can easily spot syntax errors. |
|
|
Keyword 1 token id. This can be used to mark a keyword. This should be used for general language constructs. |
|
|
Keyword 2 token id. This can be used to mark a keyword. This should be used for preprocessor commands, or variables. |
|
|
Keyword 3 token id. This can be used to mark a keyword. This should be used for data types. |
|
|
Label token id. This can be used to mark labels (eg, C mode uses this to mark ...: sequences) |
|
|
The length of this token. |
|
|
Literal 1 token id. This can be used to mark a string literal (eg, C mode uses this to mark "..." literals) |
|
|
Literal 2 token id. This can be used to mark an object literal (eg, Java mode uses this to mark true, false, etc) |
|
|
The next token in the linked list. |
|
|
Normal text token id. This should be used to mark normal text. |
|
|
Operator token id. This can be used to mark an operator. (eg, SQL mode marks +, -, etc with this token type) |
1.4.5