Text formats

Four standard types hold text with structure:

• Html is HTML that is always safe to put in a page.
• Markdown is CommonMark text.
• RichText is a structured document of blocks and marks, as a rich text editor holds one.
• LocalisedText maps locales to text, and is read with text.pick.

None of them is personal data, and none has a shape.

A food blog turns a post written in Markdown into its page, keeps only the safe part of a profile pasted from a rich text editor, and shows the words of a post in a notification:

type FormText = Text {}type PreviewText = Text {}/// A post's body as its author wrote it, made into the page's HTML.function post_body(post_markdown: Markdown) -> Html {    return markup.to_html(post_markdown)}/// A profile's about section, pasted from a rich text editor: whatever it/// holds, only the safe part is kept.function about_section(pasted_about: FormText) -> Html {    return Html(pasted_about)}/// The words a notification shows from a post, without its markup.function preview(post_page: Html) -> PreviewText {    return PreviewText(markup.to_plain_text(post_page))}

test "a post's Markdown becomes safe HTML" {    assert post_body("Soup of the **day**") == "<p>Soup of the <strong>day</strong></p>\n"    assert post_body(        "[menu](https://example.com/menu) or [prize](javascript:steal())"    ) == "<p><a href=\"https://example.com/menu\" rel=\"noopener noreferrer nofollow\">menu</a> or <a rel=\"noopener noreferrer nofollow\">prize</a></p>\n"}test "pasted HTML keeps only what is safe" {    assert about_section("<p onclick=\"steal()\">Chef</p><script>steal()</script>") == "<p>Chef</p>"    assert preview(        about_section("<h2>Menu</h2><ul><li>Soup</li><li>Bread</li></ul>")    ) == "Menu\nSoup\nBread"}

Html is safe whatever it is built from

An Html value holds only the markup its construction keeps: no scripts, no event handlers, and links only to https: and mailto:. It is a security boundary. Every way text becomes Html runs the one sanitiser: Html(raw), text.parse(raw, as: Html), a request's JSON as it is read, and markup.to_html. None of them fails, because whatever the text holds, only the safe part is kept. So none needs an attempt, and text.parse is never malformed.

A type built on Html keeps its own rules, counted on what is kept: its max_length counts the text once the script is gone. Naming format in its attempt is failure.rule_cannot_break.

A literal written where an Html goes must already be what the sanitiser keeps, since a literal is checked while compiling. That way code never holds HTML that changes when it runs, and a test compares against what is really kept. Any other literal is type.invalid_literal, whose reason and next: give the kept form: "Tom & Jerry" is written "Tom & Jerry".

A plain Text never becomes an Html without Html(...), which is the sanitiser, and text operations on an Html give plain Text. So the only Html a template may write out unescaped (Email) is one made safe. An Html and a Markdown are each sent and stored as their text. An Html read back from storage is the text that was stored, made safe when it was built.

What the sanitiser keeps

The sanitiser is ammonia, an allow-list over html5ever, which parses HTML as browsers do. Beyond the three rules above, it keeps as little as it can. It keeps these elements, with only the attributes in brackets: a (href, title), abbr (title), b, blockquote, br, code, del, em, h1 to h6, hr, i, ins, kbd, li, mark, ol (start), p, pre, q, s, small, strong, sub, sup, u and ul. It keeps no other element or attribute: no images (a remote image tells its server who read the page), tables, forms, frames, media, styles, classes or ids.

A link without its own https: or mailto: scheme loses its href: a relative link, a protocol-relative //host, http:, and javascript: however it is encoded. Every link gets rel="noopener noreferrer nofollow", whatever it was written with. Comments go.

A script, style, svg, math, template, iframe, object, embed, noscript, noembed, noframes, textarea, title, xmp, plaintext, select, applet, frame or frameset goes with everything inside it. Any other element the sanitiser doesn't allow goes but leaves its text.

What the sanitiser keeps is a fixed point: its output, sanitised again, must come back unchanged. That way a browser parsing the page builds the tree the allow-list kept, which is the defence against mutation XSS. Output that doesn't come back unchanged is kept as escaped text instead, and as nothing when even that doesn't. Markup nested more than 512 elements deep, Blink's limit, is kept as text too: html5ever parses deep nesting in quadratic time, so one large value could stall a program.

Markdown

A Markdown value is CommonMark 0.31.2, rendered by pulldown-cmark with none of its extensions: no tables, strikethrough, task lists or footnotes. It is any text with no control character but a tab, a line feed or a carriage return, so a form's text area, whose lines end in CRLF, sends Markdown as it is.

markup.to_html renders raw HTML inside the Markdown as CommonMark says, then sanitises the whole result, so raw HTML is never passed through unsanitised. An image is shown as its description, since Html keeps no images.

Plain text from HTML

markup.to_plain_text sanitises, then puts each block (a paragraph, a heading, a list item, a quote, a code block, a rule) on its own line and turns <br> into a line break. It runs white space together as a browser shows it, trims each line and leaves out empty ones. Inside <pre> it keeps the lines as written, less blank ones at either end. A list item gets no bullet or number.

text.format shows the text people read: an Html value's markup.to_plain_text, and a Markdown text's once rendered. An Html value's text.from is the HTML itself.

RichText

A RichText is a document as a rich text editor holds one, in Portable Text's flat shape with its default schema. It is a list of blocks, each holding spans of text. A block is a paragraph, a heading of level 1 to 6, a quote, or a bulleted or numbered list item with its level, 1 to 9. There is no code block, rule, image or table.

A span has its marks and at most one link. The marks are strong, emphasis, code, underline and strike_through: Portable Text's default decorators, with its em and strike-through renamed. An HTML link can't hold another, so the link is a part of the span, not a mark.

Nothing nests, so no type refers to itself. A list item's level says how deep it sits, as Portable Text's level does and as Word and Google Docs nest lists.

Its parts are three standard types, since a part must be a standard type to be read, sent and stored: RichTextBlock, a choice; RichTextSpan, an object sent as its members; and RichTextMark, an enum.

It is sent in Jadpo's own form, not with Portable Text's _type, _key, children and markDefs. The form is an array of its blocks, each in the tagged form an enum variant with fields takes in JSON (Enums). Each span is an object of text, marks and link, leaving out the link when it has none:

[  {"tag": "heading", "level": 2, "spans": [{"text": "Method", "marks": []}]},  {"tag": "paragraph", "spans": [{"text": "our mill", "marks": ["emphasis"], "link": "https://example.com/mill"}]}]

Every block and mark is an element the sanitiser keeps. A link must be one the sanitiser keeps as it is, which is either:

• an https: URL whose host is ASCII labels of letters, digits and single hyphens, the last beginning with a letter (never an IP address or punycode), with a port up to 65535; or
• a mailto: address,

each in the characters a URL writes unescaped. That refuses some links the sanitiser would keep, such as a host written in Unicode, and none it would drop; a test of random links holds the two together.

A link that isn't one of these is refused where the document is built or read, breaking pattern, rather than dropped. A literal that breaks it is type.invalid_literal, which says the rule in words rather than as its pattern. A span's text is never empty, and has no control character but a tab or a line feed.

markup.rich_to_html writes <p>, <h1> to <h6>, <blockquote>, list items in <ul> or <ol> (a deeper item in a list inside the item before it), <strong>, <em>, <code>, <u> and <s>, links with rel="noopener noreferrer nofollow", a line feed as <br>, and text escaped as the sanitiser's serialiser escapes it. It writes no white space between elements. A span's marks are written once each, in one nesting order, strong outermost and strike_through innermost, inside the link, whatever order the span lists them in. So it writes nothing the sanitiser would change, which tests hold it to, and it sanitises the result all the same: an Html is only ever what the sanitiser keeps.

A recipe site keeps each method as its editor sends it:

type EditorJson = Text {}type MethodUnreadable = Failure {    kind: FailureKind.invalid_value}/// A recipe's method as the editor sent it: its blocks and marks, with/// links only to `https:` and `mailto:` addresses.function method_from(sent: EditorJson) fails MethodUnreadable -> RichText {    return attempt json.read(sent, as: RichText) {        malformed: MethodUnreadable        mismatch: MethodUnreadable    }}/// The method on the recipe's page.function method_page(recipe_method: RichText) -> Html {    return markup.rich_to_html(recipe_method)}

test "a recipe's method becomes safe HTML" {    var oat_method: RichText = attempt method_from(        "[{\"tag\":\"heading\",\"level\":2,\"spans\":[{\"text\":\"Method\",\"marks\":[]}]},{\"tag\":\"numbered_item\",\"level\":1,\"spans\":[{\"text\":\"Soak the oats\",\"marks\":[]}]},{\"tag\":\"numbered_item\",\"level\":1,\"spans\":[{\"text\":\"Bake for \",\"marks\":[]},{\"text\":\"40 minutes\",\"marks\":[\"strong\"]}]},{\"tag\":\"paragraph\",\"spans\":[{\"text\":\"Flour from \",\"marks\":[]},{\"text\":\"our mill\",\"marks\":[\"emphasis\"],\"link\":\"https://example.com/mill\"}]}]"    )    assert method_page(        oat_method    ) == "<h2>Method</h2><ol><li>Soak the oats</li><li>Bake for <strong>40 minutes</strong></li></ol><p>Flour from <a href=\"https://example.com/mill\" rel=\"noopener noreferrer nofollow\"><em>our mill</em></a></p>"    assert text.format(        oat_method,        locale: Locale.en_gb    ) == "Method\nSoak the oats\nBake for 40 minutes\nFlour from our mill"}test "a link goes only to https: and mailto:" {    assert method_from(        "[{\"tag\":\"paragraph\",\"spans\":[{\"text\":\"prize\",\"marks\":[],\"link\":\"javascript:steal()\"}]}]"    ) fails MethodUnreadable    var note: RichText = RichText(        [            RichTextBlock.paragraph {                spans: [                    RichTextSpan {                        text: "Write to us"                        marks: []                        link: "mailto:hello@example.com"                    }                ]            }        ]    )    assert method_page(        note    ) == "<p><a href=\"mailto:hello@example.com\" rel=\"noopener noreferrer nofollow\">Write to us</a></p>"}

LocalisedText

A LocalisedText maps locales to text, such as a product's name in each language a shop sells in. It is a named map, Map<Locale, Text>, and may be empty. It is sent as JSON-LD sends a language map: an object keyed by each locale's tag, {"en-GB": "Rye loaf", "fr-FR": "Pain de seigle"}, rather than a list of entries, so each locale has one entry by construction.

Code builds one from a map, LocalisedText(map.with(...)), as it builds any named map of your application's, since there is no map literal with entries (Named collection construction).

Its keys are Locales like any other. One read from a request or from JSON is a locale your application supports: an unsupported tag breaks variant, and OpenAPI lists only the supported tags as its keys. One written in code may be any locale.

text.pick chooses an entry by the fallback chain Locales describes: the locale's own entry, else one in its language (the locale CLDR's likely subtags give for the language first, then in the canonical order), else the entry of the application's default locale. Text with none of those gives its first entry in the canonical order, so text that has any entry always shows one. An empty one gives "".

text.pick's locale: is checked as text.format's is, since it chooses text for a person. One written out must be a locale the application supports (type.locale_unsupported), and an application with no locales { } is type.temporal_locales_required:

application {}locales {    default: Locale.en_gb    supported: [Locale.en_gb, Locale.en_us, Locale.fr_fr, Locale.fr_ca, Locale.de_de]}

type ShelfName = Text {}/// A loaf's name in the shopper's language: their locale's entry, else/// one in their language, else the shop's default locale's.action shelf_name(name: LocalisedText, shopper: Locale) -> ShelfName {    return ShelfName(text.pick(name, locale: shopper))}

test "a loaf's name in the shopper's language" {    var rye: LocalisedText = LocalisedText(        map.with(map.with([], Locale.fr_fr, "Pain de seigle"), Locale.en_gb, "Rye loaf")    )    assert shelf_name(rye, Locale.fr_fr) == "Pain de seigle"  // its own entry    assert shelf_name(rye, Locale.fr_ca) == "Pain de seigle"  // its language's    assert shelf_name(rye, Locale.de_de) == "Rye loaf"  // the default locale's    var roggen: LocalisedText = LocalisedText(map.with([], Locale.de_de, "Roggenbrot"))    assert shelf_name(roggen, Locale.fr_fr) == "Roggenbrot"  // the only one it has}

Showing, sending and storing rich and localised text

text.format shows a RichText, a block or a span as the text people read in it (its HTML's markup.to_plain_text), each block on its own line. It shows a LocalisedText as text.pick gives it in that locale. Neither type has a single text form, so text.from and text.parse refuse them, and neither is ordered.

Each is a named collection the standard library declares, as a standard type may be a list or a map of parts. It is sent as JSON's own collection, and a RichText, a RichTextBlock and a LocalisedText are each stored as that JSON in one jsonb column.