SuperPi.Web.Fetch (super_pi v0.1.0)

Copy Markdown View Source

The HTTP side of web_fetch: one GET with the request-forgery policy, a decoded-byte cap and a deadline. It returns bytes and metadata. SuperPi.Web.Extract turns them into text.

Guarantees

  • Policy on every hop. SuperPi.Web.Policy.check_url/2 runs before the first request and again for each redirect. Redirects are followed here, never by Req (redirect: false), at most Policy.max_redirects/0.
  • The cap counts decoded bytes. The response streams with Req's decoding off (raw: true), and this module inflates gzip and deflate itself, a bounded step at a time (:zlib.safeInflate/2). It stops as soon as the decoded size passes max_bytes, so a 1 MB gzip bomb costs max_bytes of memory, not gigabytes. A Content-Length check would bound nothing here. max_bytes defaults to 2 MiB and is clamped to 8 MiB. A body that passes the cap is cut there and flagged body_truncated.
  • Isolation. The whole chain (DNS, requests, inflate) runs under SuperPi.Env.SandboxRunner.run_isolated/2: a wall-clock deadline and a heap kill. The machine's own tool task sets no heap limit, so this call must set it.
  • Own connection pool. Requests go through the SuperPi.WebFinch pool, not the provider pool, so agent-controlled egress shares no pool or failure budget with LLM traffic.
  • No caller headers. The request headers are fixed here. There is no way to add one.
  • Early content-type refusal. Unless accept_binary: true, a PDF or other binary response is refused from its headers, before the body is read.

Only Accept-Encoding: gzip, deflate is offered, so a server has no reason to answer with br or zstd. If it does, the fetch fails with an :encoding error.

Options

:max_bytes, :timeout_ms (whole chain, default 20 s), :accept_binary, :resolver, :pin_ip (see SuperPi.Web.Policy, off by default and unverified), :finch (pool name), :req_options (merged into every request, for operators and tests: a :plug stub keeps a test offline) and :unsafe_allow_local (test only).

Summary

Types

error()
response()

Functions

get(url, opts \\ [])

Fetches a URL. See the moduledoc for options and guarantees.

limits()

Default and hard limits, in bytes.

Types

error()

@type error() ::
  {:refused, String.t()}
  | {:dns, term()}
  | {:content_type, String.t()}
  | {:encoding, String.t()}
  | {:too_many_redirects, pos_integer()}
  | {:timeout, pos_integer()}
  | {:transport, String.t()}
  | {:crashed, String.t()}

response()

@type response() :: %{
  final_url: String.t(),
  status: non_neg_integer(),
  content_type: String.t() | nil,
  body: binary(),
  body_truncated: boolean(),
  bytes: non_neg_integer(),
  chain: [map()],
  ip: String.t() | nil,
  duration_ms: non_neg_integer()
}

Functions

get(url, opts \\ [])

@spec get(term(), keyword()) :: {:ok, response()} | {:error, error()}

Fetches a URL. See the moduledoc for options and guarantees.

limits()

@spec limits() :: %{default: pos_integer(), hard: pos_integer()}

Default and hard limits, in bytes.