Unicode, Character Sets and Text Encodings

The essential rule

Text stored as bytes cannot be interpreted reliably without knowing its encoding. Unicode describes characters; encodings describe how Unicode code points become bytes. Confusing these is the source of corrupted text and “mojibake”.

Core terminology

Character

A unit of writing or text. What a user sees as one character may contain multiple Unicode code points.

Code point

A number assigned by Unicode, written U+XXXX (for example, A is U+0041 and € is U+20AC).

Code unit

A single storage unit in a particular encoding: 8 bits in UTF-8, 16 bits in UTF-16, 32 bits in UTF-32.

Encoding

A scheme for representing code points as code units or bytes, such as UTF-8 or UTF-16.

Grapheme cluster

A user-perceived character that may contain multiple code points, such as a base letter plus a combining accent.

Why older code pages caused problems

ASCII defines 128 values (0–127), including English letters, digits, punctuation and control characters. Historic 8-bit encodings reused 128–255 for different alphabets and symbols; the same byte could represent different text under different code pages. Double-byte character sets added variable-width interpretation for some scripts. Sending bytes without their encoding therefore made text ambiguous.

Unicode is not an encoding

Unicode gives text a shared numbering system independent of fonts, operating systems and legacy code pages. For example, Hello is U+0048 U+0065 U+006C U+006C U+006F. Those numbers still need an encoding before they can be saved or transmitted.

Common encodings

Byte order marks (BOMs)

A BOM can identify encoding and byte order, particularly for UTF-16 or UTF-32. UTF-8 does not require a BOM; use it only where interoperability requirements call for one. Modern HTML should declare <meta charset="utf-8"> and be served with the correct HTTP Content-Type charset.

Practical rules

1. Choose UTF-8 by default for files, APIs and interchange formats unless a specification dictates otherwise.
2. Specify the encoding when writing or reading files, network content and database exports. Never rely on a machine’s default code page.
3. Convert at the boundary, using the actual source encoding. Do not relabel bytes as UTF-8 and expect them to become valid Unicode.
4. Handle conversion errors deliberately; silent replacement with ? or U+FFFD may irreversibly lose the original data.
5. Do not equate byte count, code-unit count, code-point count and visible-character count.
6. Use Unicode-aware libraries for searching, case conversion, normalization, sorting and text boundaries; behaviour depends on language and context.
7. When debugging corrupted text, inspect the original bytes and identify every decode/encode step before applying a fix.

Worked example

A (U+0041) is 41 in UTF-8 and 41 00 in UTF-16 little-endian. € (U+20AC) is E2 82 AC in UTF-8 and AC 20 in UTF-16 little-endian. If UTF-8 bytes are decoded as Windows-1252, the euro sign can appear as €: the bytes did not necessarily change; the interpretation did.

Delphi / Windows note

In modern Unicode Delphi, string is UTF-16-based; Length counts UTF-16 code units, not grapheme clusters. For interchange, use explicit UTF-8 conversion (for example, TEncoding.UTF8), and be explicit when interoperating with legacy AnsiString data or Windows ANSI APIs.

Historical perspective and attribution

This reference condenses the concepts explained in Joel Spolsky’s The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) (8 October 2003). The original article uses historical examples and includes some details subsequently superseded by Unicode and UTF-8 standards. This reference incorporates modern clarifications and is not a verbatim reproduction.