Runes

UTF-8 rune (Unicode code point) methods. More...

Collaboration diagram for Runes:

Data Structures
struct	sys_rune_tokenize_t
	Tokenizer state, grouping a stream into maximal runs of same-class runes. More...

Macros
#define	RUNE_ERROR   ((rune_t)0xFFFD)
	Sentinel value for a malformed UTF-8 sequence. More...

Typedefs
typedef struct sys_rune_tokenize_t	sys_rune_tokenize_t
	Tokenizer state, grouping a stream into maximal runs of same-class runes. More...

Enumerations
enum	sys_rune_class_t {   sys_rune_other = 0, sys_rune_space, sys_rune_digit, sys_rune_alpha,   sys_rune_punct, sys_rune_symbol, sys_rune_control }
	Classification of a rune, or of a maximal run of same-class runes. More...

Functions
const char *	sys_rune_next (const char *str, rune_t *rune)
	Decode the next UTF-8 rune from a string. More...
size_t	sys_rune_count (const char *str)
	Count the runes in a UTF-8 string. More...
bool	sys_rune_valid (const char *str)
	Check whether a string is well-formed UTF-8. More...
static bool	sys_rune_is_digit (rune_t r)
	Reports whether r is a decimal digit.
static bool	sys_rune_is_space (rune_t r)
	Reports whether r is a space character (space, \t, \n, \v, \f, \r, U+0085 NEL, or U+00A0 NBSP).
static bool	sys_rune_is_upper (rune_t r)
	Reports whether r is an uppercase letter.
static bool	sys_rune_is_lower (rune_t r)
	Reports whether r is a lowercase letter.
static bool	sys_rune_is_alpha (rune_t r)
	Reports whether r is a letter.
static rune_t	sys_rune_to_upper (rune_t r)
	Converts r to its uppercase form, if it has one. More...
static rune_t	sys_rune_to_lower (rune_t r)
	Converts r to its lowercase form, if it has one. More...
static bool	sys_rune_is_punct (rune_t r)
	Reports whether r is a punctuation character. More...
static bool	sys_rune_is_symbol (rune_t r)
	Reports whether r is a symbol character (math, currency, or modifier symbols). More...
static bool	sys_rune_is_control (rune_t r)
	Reports whether r is a control character.
sys_rune_class_t	sys_rune_isa (rune_t r)
	Classify a single rune. More...
sys_rune_tokenize_t	sys_rune_tokenize_init (sys_iostream_t *stream)
	Initialize a tokenizer over a stream. More...
bool	sys_rune_tokenize_next (sys_rune_tokenize_t *it)
	Advance to the next maximal run of same-class runes. More...
size_t	sys_rune_tokenize_token (sys_rune_tokenize_t *it, char *buf, size_t cap)
	Read the current token's text. More...

Detailed Description

UTF-8 rune (Unicode code point) methods.

Macro Definition Documentation

RUNE_ERROR

#define RUNE_ERROR   ((rune_t)0xFFFD)

Sentinel value for a malformed UTF-8 sequence.

This is the Unicode replacement character (U+FFFD), returned in rune when the bytes at the current position do not form a valid UTF-8 sequence. The returned pointer still advances (past the offending byte), so iteration can continue.

Definition at line 39 of file rune.h.

Typedef Documentation

sys_rune_tokenize_t

typedef struct sys_rune_tokenize_t sys_rune_tokenize_t

Tokenizer state, grouping a stream into maximal runs of same-class runes.

Reads from the sys_iostream_t given to sys_rune_tokenize_init(). Records only where the current token starts and how long it is - the tokenizer itself never allocates or limits a token's length (whether that holds for the stream as a whole depends on its backend). Use sys_rune_tokenize_token() to read its actual text.

Enumeration Type Documentation

sys_rune_class_t

enum sys_rune_class_t

Classification of a rune, or of a maximal run of same-class runes.

Enumerator
sys_rune_other	unclassified - also covers RUNE_ERROR
sys_rune_space	see sys_rune_is_space()
sys_rune_digit	see sys_rune_is_digit()
sys_rune_alpha	see sys_rune_is_alpha()
sys_rune_punct	see sys_rune_is_punct()
sys_rune_symbol	see sys_rune_is_symbol()
sys_rune_control	see sys_rune_is_control()

Definition at line 303 of file rune.h.

             {
  sys_rune_other = 0, 
  sys_rune_space,      
  sys_rune_digit,      
  sys_rune_alpha,      
  sys_rune_punct,      
  sys_rune_symbol,     
  sys_rune_control,    
} sys_rune_class_t;

Function Documentation

sys_rune_count()

size_t sys_rune_count

(

const char *

str

)

Count the runes in a UTF-8 string.

Parameters

str	Pointer to a null-terminated UTF-8 string, or NULL.

Returns

The number of runes decoded before the terminator (each malformed byte sequence counts as one), or 0 if str is NULL or empty. Use sys_rune_valid() to check well-formedness.

sys_rune_is_punct()

static bool sys_rune_is_punct ( rune_t  r )

inlinestatic

Reports whether r is a punctuation character.

Follows Unicode's Punctuation category, not the C locale's broader ispunct() - math/currency/modifier symbols are sys_rune_is_symbol() instead.

Definition at line 202 of file rune.h.

                                               {
  switch (r) {
  case '!':
  case '"':
  case '#':
  case '%':
  case '&':
  case '\'':
  case '(':
  case ')':
  case '*':
  case ',':
  case '-':
  case '.':
  case '/':
  case ':':
  case ';':
  case '?':
  case '@':
  case '[':
  case '\\':
  case ']':
  case '_':
  case '{':
  case '}':
  case 0x00A1: // inverted exclamation mark
  case 0x00A7: // section sign
  case 0x00B6: // pilcrow sign
  case 0x00B7: // middle dot
  case 0x00BF: // inverted question mark
    return true;
  default:
    return false;
  }
}

sys_rune_is_symbol()

static bool sys_rune_is_symbol ( rune_t  r )

inlinestatic

Reports whether r is a symbol character (math, currency, or modifier symbols).

The complement of sys_rune_is_punct() within the printable ASCII/Latin-1 range - together they partition it.

Definition at line 246 of file rune.h.

                                                {
  switch (r) {
  case '$':
  case '+':
  case '<':
  case '=':
  case '>':
  case '^':
  case '`':
  case '|':
  case '~':
  case 0x00A2: // cent sign
  case 0x00A3: // pound sign
  case 0x00A4: // currency sign
  case 0x00A5: // yen sign
  case 0x00A6: // broken bar
  case 0x00A8: // diaeresis
  case 0x00A9: // copyright sign
  case 0x00AC: // not sign
  case 0x00AE: // registered sign
  case 0x00AF: // macron
  case 0x00B0: // degree sign
  case 0x00B1: // plus-minus sign
  case 0x00B4: // acute accent
  case 0x00B8: // cedilla
  case 0x00D7: // multiplication sign
  case 0x00F7: // division sign
    return true;
  default:
    return false;
  }
}

sys_rune_isa()

sys_rune_class_t sys_rune_isa

(

rune_t

r

)

Classify a single rune.

Parameters

r	The rune to classify.

Returns

The first matching classification (space, control, digit, alpha, punct, symbol, in that order), or sys_rune_other if none match. '\t' '\n' '\v' '\f' '\r' and NEL match both sys_rune_is_space() and sys_rune_is_control() - space wins, so sys_rune_control ends up covering only genuinely non-whitespace control codes.

sys_rune_next()

const char* sys_rune_next	(	const char *	str,
		rune_t *	rune
	)

Decode the next UTF-8 rune from a string.

Parameters

str	Pointer to a null-terminated UTF-8 string.
rune	Pointer to store the decoded rune. Set to RUNE_ERROR on a malformed sequence, or 0 if the string is exhausted.

Returns

Pointer to the byte following the consumed rune (pass this back in as str for the next call), or NULL once the string is exhausted. On a malformed sequence, the pointer still advances - by exactly one byte - so a decode loop can keep going after an error.

sys_rune_to_lower()

static rune_t sys_rune_to_lower ( rune_t  r )

inlinestatic

Converts r to its lowercase form, if it has one.

Parameters

r	The rune to convert.

Returns

The lowercase form of r, or r unchanged if it has none.

Definition at line 181 of file rune.h.

                                                 {
  if (r >= 'A' && r <= 'Z') {
    return r + 32;
  }
  if (r >= 0x00C0 && r <= 0x00D6) { // A grave - O diaeresis
    return r + 0x20;
  }
  if (r >= 0x00D8 && r <= 0x00DE) { // O stroke - Thorn
    return r + 0x20;
  }
  return r;
}

sys_rune_to_upper()

static rune_t sys_rune_to_upper ( rune_t  r )

inlinestatic

Converts r to its uppercase form, if it has one.

Parameters

r	The rune to convert.

Returns

The uppercase form of r, or r unchanged if it has none (this includes sharp s and y diaeresis, which have no single-codepoint uppercase form).

Definition at line 162 of file rune.h.

                                                 {
  if (r >= 'a' && r <= 'z') {
    return r - 32;
  }
  if (r >= 0x00E0 && r <= 0x00F6) { // a grave - o diaeresis
    return r - 0x20;
  }
  if (r >= 0x00F8 && r <= 0x00FE) { // o stroke - thorn
    return r - 0x20;
  }
  return r;
}

sys_rune_tokenize_init()

sys_rune_tokenize_t sys_rune_tokenize_init

(

sys_iostream_t *

stream

)

Initialize a tokenizer over a stream.

Parameters

stream

The stream to read runes from, or NULL.

Returns

An initialized tokenizer, positioned before the first token.

sys_rune_tokenize_next()

bool sys_rune_tokenize_next

(

sys_rune_tokenize_t *

it

)

Advance to the next maximal run of same-class runes.

Parameters

it	Pointer to the tokenizer state.

Returns

true if a token was found (it->start, it->bytes, it->runes and it->isa are populated), false if the stream is exhausted.

sys_rune_tokenize_token()

size_t sys_rune_tokenize_token	(	sys_rune_tokenize_t *	it,
		char *	buf,
		size_t	cap
	)

Read the current token's text.

Parameters

it	The tokenizer, positioned at a token by a prior successful sys_rune_tokenize_next() call.
buf	Destination buffer.
cap	Capacity of buf.

Returns

The number of bytes copied - min(it->bytes, cap); cap == 0 or too small silently truncates rather than failing.

sys_rune_valid()

bool sys_rune_valid

(

const char *

str

)

Check whether a string is well-formed UTF-8.

Parameters

str	Pointer to a null-terminated UTF-8 string, or NULL.

Returns

true if every byte sequence up to the terminator decodes without error, or if str is NULL or empty; false if any malformed sequence is found.