# `SuperPi.Web.Extract`
[🔗](https://github.com/oleksandr/super-pi-beam/blob/v0.1.0/lib/super_pi/web/extract.ex#L1)

Turns a fetched body into text a model can read: markdown for HTML, pretty JSON, plain text.

Guarantees, for any input bytes:

- it never raises (a failure falls back to the decoded text with tags stripped);
- the text is valid UTF-8, with control characters removed;
- `page/3` and the `:max_chars` option never return more than `max_chars` characters. A
  character is a Unicode code point, counted the same way everywhere (`char_count/1`).

## HTML

The tree is parsed with Floki. `script`, `style`, `noscript`, `template`, `svg`, `iframe`,
`nav`, `footer`, `aside`, `button`, hidden elements and ARIA navigation and footer
landmarks are dropped. The extraction root is the first `<main>`, else all `<article>`
elements, else `<body>`. Without `<main>` or `<article>` the page `<header>` is dropped too.
The markdown covers headings, paragraphs, lists, links, emphasis, code, blockquotes, tables
and images that have alt text. The `text` format keeps the same structure without markup.

Parsing a large page is the expensive step. The caller runs `extract/3` under
`SuperPi.Env.SandboxRunner.run_isolated/2` for a deadline and a heap limit.

## Other types

JSON is pretty-printed (falling back to the text if it does not parse). PDF is not extracted:
`kind/2` reports `:pdf` and the caller refuses it with a message that names the `artifact`
argument. Other text types pass through.

## Character sets

The body is decoded from the BOM, the `Content-Type` charset or an HTML `<meta charset>`.
UTF-8, UTF-16, ISO-8859-1 and Windows-1252 are converted. Any other charset is read as UTF-8
and invalid bytes become U+FFFD.

# `kind`

```elixir
@type kind() :: :html | :json | :text | :pdf | :binary | :unknown
```

# `result`

```elixir
@type result() :: %{
  kind: kind(),
  title: String.t() | nil,
  text: String.t(),
  links: [String.t()],
  chars: non_neg_integer(),
  offset: non_neg_integer(),
  next_offset: non_neg_integer() | nil,
  truncated: boolean()
}
```

# `char_count`

```elixir
@spec char_count(binary()) :: non_neg_integer()
```

Number of code points in a valid UTF-8 binary.

# `extract`

```elixir
@spec extract(binary(), String.t() | nil, keyword()) :: result()
```

Extracts text from a body.

Options: `:format` (`"markdown"` default, `"text"` or `"raw"`), `:base_url` (resolves relative
links), `:offset` and `:max_chars` (page the text, see `page/3`; without `:max_chars` the whole
text is returned).

# `kind`

```elixir
@spec kind(String.t() | nil, binary() | nil) :: kind()
```

Classifies a response by `Content-Type`, and by a prefix of the body when the type says
nothing. `body` may be `nil` to classify on the header alone: a missing or generic type then
gives `:unknown`.

# `page`

```elixir
@spec page(String.t(), non_neg_integer(), pos_integer()) :: {String.t(), map()}
```

Returns the page of `text` that starts at code point `offset` and holds at most `max_chars`
code points, with the continuation offset. `text` must be valid UTF-8.

---

*Consult [api-reference.md](api-reference.md) for complete listing*