Request-forgery policy for web_fetch: which URLs an agent may make the node request.
The model controls the URL, so the node must not become a proxy into its own network. The rule is deny by default, allow global unicast only.
check_url/2 refuses a URL when:
- the scheme is not
httporhttps, or the URL has userinfo, a backslash or whitespace; - the port is not 80 or 443;
- the host is
localhost,*.localhost,*.local,*.internalormetadata.google.internal, whatever it resolves to; - the host is a numeric form that does not parse as one clear address (
0x7f.1,2130706433are checked as the address they mean); - ANY address the host resolves to is reserved (
reserved?/1).
reserved?/1 covers loopback, private, link-local (including 169.254.0.0/16, the cloud
metadata endpoint), CGNAT, multicast, unspecified, documentation and benchmarking ranges, and
for IPv6 everything outside 2000::/3 (ULA fc00::/7, link-local fe80::/10 and so on).
IPv4-mapped (::ffff:a.b.c.d), NAT64 (64:ff9b::/96) and 6to4 (2002::/16) forms are
unwrapped and the embedded IPv4 address is checked, so a form is refused exactly when its
unwrapped address is.
Redirects are followed by the caller, never by the HTTP client: at most max_redirects/0
hops, and check_url/2 runs on every hop (redirect_url/2 resolves a Location value).
A public host that redirects to 127.0.0.1 is refused at the second hop.
The module is pure except for the default resolver. Tests and callers inject :resolver
(fn host -> {:ok, [ip]} | {:error, reason} end) so no check touches DNS.
Residual risk: DNS rebinding
The check resolves the name, the HTTP client resolves it again when it connects. A hostile
DNS server can answer with a public address to the first lookup and 127.0.0.1 to the
second. pin/2 closes the window by building a request against the checked address, and the
fetcher uses it when pin_ip: true. That path is OFF by default and UNVERIFIED against the
installed Req, Finch and Mint (TLS verification and the Host header need an end-to-end
check). A deployment that cannot tolerate the window should use network: "allowlist" or an
egress proxy that filters by destination address.
Test-only option
unsafe_allow_local: true skips the port, name and address checks so a test can reach a
server on 127.0.0.1. It is an option of the call and cannot come from a tool argument. It
must never be set in production config.
Summary
Functions
Checks a URL and resolves its host.
Text form of an address.
Maximum number of redirects a fetch follows.
Builds the request target for a pinned-IP connect: the URI with the checked address as host
and the original host name for the Host header, SNI and certificate check.
Resolves a Location value against the URL it came from. Never raises.
True when an address is outside global unicast space and so must never be fetched.
Default resolver: the system resolver, IPv4 and IPv6, every address returned.
Types
@type ip() :: :inet.ip_address()
Functions
@spec check_url(term(), keyword()) :: {:ok, target()} | {:error, {:refused, String.t()} | {:dns, term()}}
Checks a URL and resolves its host.
Returns the parsed URI, the normalized host and the checked addresses, or
{:error, {:refused, message}} or {:error, {:dns, reason}}.
Options: :resolver (default system_resolver/1) and :unsafe_allow_local (test only).
Text form of an address.
@spec max_redirects() :: pos_integer()
Maximum number of redirects a fetch follows.
Builds the request target for a pinned-IP connect: the URI with the checked address as host
and the original host name for the Host header, SNI and certificate check.
Unverified, see the moduledoc.
Resolves a Location value against the URL it came from. Never raises.
True when an address is outside global unicast space and so must never be fetched.
A malformed tuple counts as reserved.
Default resolver: the system resolver, IPv4 and IPv6, every address returned.