UTF-8 Binary Encoding

What Makes UTF-8 Special?

UTF-8 is the dominant character encoding on the web — over 98% of all web pages use it. The reason is elegant: it is fully backward-compatible with ASCII. Every ASCII character (0-127) is exactly 1 byte in UTF-8, identical to its ASCII binary representation. For everything else, UTF-8 uses 2-4 bytes, with clever marker bits that allow decoders to detect character boundaries without scanning backwards. I think of UTF-8 as the encoding that gives you the best of both worlds: ASCII efficiency for English and Unicode coverage for everything else.

UTF-8 uses a self-synchronizing prefix scheme. Every byte announces its role in the sequence:

• 0xxxxxxx — Single byte (ASCII). Leading 0 means "this is it, one byte only."
• 110xxxxx — Start of 2-byte sequence. The 110 prefix tells the decoder "expect 1 more byte."
• 1110xxxx — Start of 3-byte sequence. The 1110 prefix means "expect 2 more bytes."
• 11110xxx — Start of 4-byte sequence. The 11110 prefix means "expect 3 more bytes."
• 10xxxxxx — Continuation byte. Every non-leading byte starts with 10, making it unmistakably a continuation.

This design means you can jump into the middle of any UTF-8 stream and find the start of the next character in at most 3 bytes — a property called "self-synchronization."

Decoding Examples

Let me decode three characters to show how the markers work in practice:

1. 'A' (U+0041): 01000001 — Starts with 0, single byte. No marker bits to strip. Binary = 1000001 = 65 = 'A'.
2. 'ñ' (U+00F1): 11000011 10110001 — Leading byte 110 means 2-byte sequence. Strip marker: 00011 from byte 1 + 110001 from byte 2 = 00011 110001 = 11110001 = 241 = U+00F1.
3. '中' (U+4E2D): 11100100 10111000 10101101 — Leading byte 1110 means 3-byte sequence. Strip and combine: 0100 111000 101101 = 0100111000101101 = 20,013 = U+4E2D.

Encoding a code point to UTF-8 binary follows a straightforward procedure. Here are the rules for each byte count:

• U+0000 to U+007F (1 byte): Byte = 0xxxxxxx, where xxxxxxx is the 7-bit code point value. Example: U+0041 = 01000001.
• U+0080 to U+07FF (2 bytes): 110xxxxx 10xxxxxx. The code point needs 11 bits, split 5+6 across the two bytes. Example: U+00F1 = 11000011 10110001.
• U+0800 to U+FFFF (3 bytes): 1110xxxx 10xxxxxx 10xxxxxx. The code point needs 16 bits, split 4+6+6. Example: U+4E2D = 11100100 10111000 10101101.
• U+10000 to U+10FFFF (4 bytes): 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx. Needs 21 bits, split 3+6+6+6. Example: U+1F600 = 11110000 10011111 10011000 10000000.

I have written UTF-8 encoders by hand for embedded systems, and this pattern makes it simple to implement—just check the code point range, allocate the right number of bytes, fill in the marker bits, and insert the code point bits into the x positions.

UTF-8's ASCII compatibility is not accidental — it is baked into the bit pattern design. Every ASCII character has a code point from 0-127, which in binary is 0xxxxxxx (7 bits). UTF-8 treats these as single bytes where the leading bit is 0. Non-ASCII bytes always have the leading bit set to 1 (10xxxxxx, 110xxxxx, 1110xxxx, 11110xxx).

This means any UTF-8 decoder that encounters byte 01000001 knows it is ASCII 'A'. There is no ambiguity. And any ASCII-only tool (like old grep or sed on ASCII-only locales) can process UTF-8 text and still correctly handle all the English parts — it just skips or garbles the multi-byte sequences.

Not every byte sequence is valid UTF-8. When I am debugging encoding issues, I check for these common violations:

• Overlong sequences: Encoding an ASCII character in 2+ bytes (e.g., 11000000 10100001 for '/'). This is a security risk and must be rejected.
• Missing continuation bytes: A 1110xxxx start byte followed by a non-10xxxxxx byte means the stream is corrupted.
• Surrogate bytes in UTF-8: 11101101 10100000 10000000 and above are in the surrogate range (U+D800-U+DFFF) and are invalid UTF-8.
• Too high code point: Values above U+10FFFF (like 11111100 sequences from obsolete UTF-8 specs) are invalid.

Overlong sequences are a security-critical concept in UTF-8. An overlong sequence uses more bytes than necessary to encode a code point. For example, U+002F (the '/' character) should be the single byte 00101111. But an attacker could encode it as the 3-byte sequence 11100000 10000000 10101111.

Why does this matter? If a UTF-8 decoder accepts overlong sequences, a path validation check like if path contains "/" might fail on the ASCII check but succeed on the overlong-encoded version — the decoder then normalizes it to '/', opening up directory traversal attacks. Validating against overlong sequences is non-negotiable in any security-conscious UTF-8 implementation.

Understanding UTF-8 binary encoding is invaluable when working with binary translation tools. If you feed a multi-byte UTF-8 sequence into a standard binary to text converter that assumes 8-bit ASCII, each UTF-8 byte gets decoded as a separate character rather than being combined into one code point. The binary decoder table in our ASCII reference is designed for single-byte characters — for UTF-8, you need to group the bytes using the prefix rules described here. This is why knowing your encoding layers is essential for accurate binary-to-text conversion.