Tokenizer state, grouping a stream into maximal runs of same-class runes. More...
#include <rune.h>
Data Fields | |
| sys_iostream_t * | stream |
| source (private) | |
| ptrdiff_t | start |
| absolute stream position where the current token begins | |
| size_t | bytes |
| length of the current token, in bytes | |
| size_t | runes |
| number of runes in the current token | |
| sys_rune_class_t | isa |
| classification shared by the whole token | |
Tokenizer state, grouping a stream into maximal runs of same-class runes.
Reads from the sys_iostream_t given to sys_rune_tokenize_init(). Records only where the current token starts and how long it is - the tokenizer itself never allocates or limits a token's length (whether that holds for the stream as a whole depends on its backend). Use sys_rune_tokenize_token() to read its actual text.