# `SuperPi.Web.Fetch`
[🔗](https://github.com/oleksandr/super-pi-beam/blob/v0.1.0/lib/super_pi/web/fetch.ex#L1)

The HTTP side of `web_fetch`: one GET with the request-forgery policy, a decoded-byte cap and
a deadline. It returns bytes and metadata. `SuperPi.Web.Extract` turns them into text.

## Guarantees

- **Policy on every hop.** `SuperPi.Web.Policy.check_url/2` runs before the first request and
  again for each redirect. Redirects are followed here, never by Req (`redirect: false`), at
  most `Policy.max_redirects/0`.
- **The cap counts decoded bytes.** The response streams with Req's decoding off
  (`raw: true`), and this module inflates `gzip` and `deflate` itself, a bounded step at a
  time (`:zlib.safeInflate/2`). It stops as soon as the decoded size passes `max_bytes`, so a
  1 MB gzip bomb costs `max_bytes` of memory, not gigabytes. A `Content-Length` check would
  bound nothing here. `max_bytes` defaults to 2 MiB and is clamped to 8 MiB. A body that passes
  the cap is cut there and flagged `body_truncated`.
- **Isolation.** The whole chain (DNS, requests, inflate) runs under
  `SuperPi.Env.SandboxRunner.run_isolated/2`: a wall-clock deadline and a heap kill. The
  machine's own tool task sets no heap limit, so this call must set it.
- **Own connection pool.** Requests go through the `SuperPi.WebFinch` pool, not the provider
  pool, so agent-controlled egress shares no pool or failure budget with LLM traffic.
- **No caller headers.** The request headers are fixed here. There is no way to add one.
- **Early content-type refusal.** Unless `accept_binary: true`, a PDF or other binary
  response is refused from its headers, before the body is read.

Only `Accept-Encoding: gzip, deflate` is offered, so a server has no reason to answer with
`br` or `zstd`. If it does, the fetch fails with an `:encoding` error.

## Options

`:max_bytes`, `:timeout_ms` (whole chain, default 20 s), `:accept_binary`, `:resolver`,
`:pin_ip` (see `SuperPi.Web.Policy`, off by default and unverified), `:finch` (pool name),
`:req_options` (merged into every request, for operators and tests: a `:plug` stub keeps a
test offline) and `:unsafe_allow_local` (test only).

# `error`

```elixir
@type error() ::
  {:refused, String.t()}
  | {:dns, term()}
  | {:content_type, String.t()}
  | {:encoding, String.t()}
  | {:too_many_redirects, pos_integer()}
  | {:timeout, pos_integer()}
  | {:transport, String.t()}
  | {:crashed, String.t()}
```

# `response`

```elixir
@type response() :: %{
  final_url: String.t(),
  status: non_neg_integer(),
  content_type: String.t() | nil,
  body: binary(),
  body_truncated: boolean(),
  bytes: non_neg_integer(),
  chain: [map()],
  ip: String.t() | nil,
  duration_ms: non_neg_integer()
}
```

# `get`

```elixir
@spec get(term(), keyword()) :: {:ok, response()} | {:error, error()}
```

Fetches a URL. See the moduledoc for options and guarantees.

# `limits`

```elixir
@spec limits() :: %{default: pos_integer(), hard: pos_integer()}
```

Default and hard limits, in bytes.

---

*Consult [api-reference.md](api-reference.md) for complete listing*