SuperPi.Web.Extract (super_pi v0.1.0)

Copy Markdown View Source

Turns a fetched body into text a model can read: markdown for HTML, pretty JSON, plain text.

Guarantees, for any input bytes:

  • it never raises (a failure falls back to the decoded text with tags stripped);
  • the text is valid UTF-8, with control characters removed;
  • page/3 and the :max_chars option never return more than max_chars characters. A character is a Unicode code point, counted the same way everywhere (char_count/1).

HTML

The tree is parsed with Floki. script, style, noscript, template, svg, iframe, nav, footer, aside, button, hidden elements and ARIA navigation and footer landmarks are dropped. The extraction root is the first <main>, else all <article> elements, else <body>. Without <main> or <article> the page <header> is dropped too. The markdown covers headings, paragraphs, lists, links, emphasis, code, blockquotes, tables and images that have alt text. The text format keeps the same structure without markup.

Parsing a large page is the expensive step. The caller runs extract/3 under SuperPi.Env.SandboxRunner.run_isolated/2 for a deadline and a heap limit.

Other types

JSON is pretty-printed (falling back to the text if it does not parse). PDF is not extracted: kind/2 reports :pdf and the caller refuses it with a message that names the artifact argument. Other text types pass through.

Character sets

The body is decoded from the BOM, the Content-Type charset or an HTML <meta charset>. UTF-8, UTF-16, ISO-8859-1 and Windows-1252 are converted. Any other charset is read as UTF-8 and invalid bytes become U+FFFD.

Summary

Types

kind()
result()

Functions

char_count(text)

Number of code points in a valid UTF-8 binary.

extract(body, content_type, opts \\ [])

Extracts text from a body.

kind(content_type, body)

Classifies a response by Content-Type, and by a prefix of the body when the type says nothing. body may be nil to classify on the header alone: a missing or generic type then gives :unknown.

page(text, offset, max_chars)

Returns the page of text that starts at code point offset and holds at most max_chars code points, with the continuation offset. text must be valid UTF-8.

Types

kind()

@type kind() :: :html | :json | :text | :pdf | :binary | :unknown

result()

@type result() :: %{
  kind: kind(),
  title: String.t() | nil,
  text: String.t(),
  links: [String.t()],
  chars: non_neg_integer(),
  offset: non_neg_integer(),
  next_offset: non_neg_integer() | nil,
  truncated: boolean()
}

Functions

char_count(text)

@spec char_count(binary()) :: non_neg_integer()

Number of code points in a valid UTF-8 binary.

extract(body, content_type, opts \\ [])

@spec extract(binary(), String.t() | nil, keyword()) :: result()

Extracts text from a body.

Options: :format ("markdown" default, "text" or "raw"), :base_url (resolves relative links), :offset and :max_chars (page the text, see page/3; without :max_chars the whole text is returned).

kind(content_type, body)

@spec kind(String.t() | nil, binary() | nil) :: kind()

Classifies a response by Content-Type, and by a prefix of the body when the type says nothing. body may be nil to classify on the header alone: a missing or generic type then gives :unknown.

page(text, offset, max_chars)

@spec page(String.t(), non_neg_integer(), pos_integer()) :: {String.t(), map()}

Returns the page of text that starts at code point offset and holds at most max_chars code points, with the continuation offset. text must be valid UTF-8.