STF-7: The Silly 7-bit Transformation Format

Premise

Knowing that:

1. The zombie apocalypse could come upon us at any moment,
2. This will leave us with nothing except 7-bit teletype machines for communication,
3. We will still need to send emojis to each other,

I have taken it upon myself to create a 7-bit encoding scheme for Unicode.

Features

This encoding has a number of advantages over UTF-7:

• Self-synchronization.
• One-to-one mapping between each Unicode scalar value and its encoded representation.
• Native encoding of supplementary characters (without surrogates).
• Direct and indirect characters occupy different ASCII ranges.
• U+10FFFF encodes in six ASCII values.
• Single byte order mark (BOM).

Note that STF-7 still shares some of UTF-7's security concerns. See the dedicated section below for more information.

Encoding Steps

Some Unicode characters are encoded directly as single ASCII values. Others are encoded indirectly as multiple values, up to and including six.

Direct Encoding

The following character ranges (inclusive) are encoded directly by making the encoded ASCII value the same as the Unicode scalar value.

Directly encoded characters can be interpreted as-is by an ASCII system without STF-7 support.

Indirect Encoding

Any Unicode character not directly encoded is indirectly encoded. In this case, the scalar value is split into chunks of four bits.

The number of ASCII values needed will reflect the number of four bit chunks the scalar value will be divided into. Padding is done by prepending any necessary zeroes to the most significant chunk. Each chunk will inherently have a value between 0x0 and 0xF.

Initial Chunks

Start with the most significant chunk and work your way through each one except for the last one. Output a single ASCII value for each one in place of its four bit value according to the table below:

Closing Chunk

For a Unicode value's final chunk, use the below table instead:

After this, the next Unicode value can be encoded.

Decoding

The decoding steps are performed in reverse.

Decoders should enforce these two specific rules:

1. Any character that can be directly encoded should be directly encoded.
2. Indirectly encoded characters should be encoded using as few bytes as possible.

Any decoded Unicode value that doesn't follow these rules should be rejected.

When randomly seeking, do not immediately resume decoding unless a directly encoded value is detected. Seek immediately past the next closing chunk instead, and resume decoding there.

Byte Order Mark

STF-7's byte order mark is: 0x3A 0x2F 0x3A 0x7E.

Security Concerns

As STF-7 does not directly encode the entire ASCII range, certain single ASCII values are represented using pairs of ASCII values in the encoded output.

Consider, for instance, the hypothetical case of filtering JavaScript out of HTML. As such, we are going to be filtering the encoded text for any occurrences of <script>. We can typically do this reliably under most code pages because the Basic ASCII range (0x00-0x7F) is encoded as-is. But with STF-7, <script> is encoded as ${script$}, thereby allowing it to evade filtering. The encoded text can still be decoded later in the process and executed with every script block left in tact.

Therefore, STF-7 should only be implemented as a part of a tightly-controlled system. Encoding should happen immediately before entering a 7-bit transmission medium and decoding should happen immediately after exiting this medium.

Comparison

The below table compares how many bytes STF-7 requires for different scalar values compared to other encodings. UTF-7 is not listed as it is rather freeform and arbitrary.