Direct page fetches

When Oda reads a page for a task, one of the readers set up for your deployment does the reading. The built-in one opens public pages directly from this computer and sends no credentials; this page is the reference for its safety rules: which addresses it accepts, which destinations it refuses, the redirects it follows, and the limits it keeps. The calls that ask for a read, and the texts they return, are on Web search and page fetch; how your deployment chooses a reader is on Web search providers.

What the direct reader is

Which addresses it accepts

Which destinations it refuses

Redirects

The limits it keeps

When a read fails

Questions, answered

What happens if a name resolves to several addresses and one is not public?

Oda checks the whole set. The connection is opened only to the addresses that passed, and if any address in the set is not a public internet address, the read is refused as a whole: URL hostname "<hostname>" resolves to a non-public IP address. This is what keeps internal services out of reach of a direct read.

Can the destination change after Oda has checked it?

No. Oda resolves the name once, checks every address that comes back, and opens the connection only to that checked set, so a name cannot resolve to a private address in the middle of a read.

What does the NAT64 address message mean?

Some networks translate a name from one address family to another. When that translation lands on a private address, Oda refuses the read like any other non-public destination: URL hostname "<hostname>" resolves through NAT64 to a non-public IPv4 address. There is nothing to correct on your side; the destination stays out of reach for a direct read.

Is a private address allowed when Oda runs behind a proxy?

No. A private or loopback address written as numbers is refused even on the proxy path: it is never handed to the proxy. The same addresses are refused when no proxy is set.

Which redirects does a direct fetch follow?

Only redirects that stay on the same site: the same scheme, the same host and the same port (the 301, 302, 303, 307 and 308 responses). Each new address is checked again before it is opened, and a chain longer than the bound (five by default) is refused. A redirect to another site is never followed: cross-origin redirect to <origin> is not followed automatically; retry against that URL directly.

What if a redirect response has no destination?

The read fails before anything further is fetched: redirect response (HTTP <status>) without a Location header. Nothing was read from that address. Try again, or give Oda the page's final address if you know it.

Why is a large page cut short instead of refused?

The size check works two ways. A response that announces a size above the limit is refused before its body is downloaded; a body that keeps growing past the limit while it arrives is cut short at the limit, and the result is marked as truncated. A body that arrives exactly at the limit is not marked.

Which page kinds can a direct fetch read?

Text-carrying pages: HTML, plain text, JSON and XML. A PDF, an image or another file kind is refused: unsupported content type "<type>", and a page that names no content type is refused the same way. A page is decoded with the character set the response declares, and a set Oda does not recognise is refused rather than guessed at.

What does "web fetch failed: ..." mean?

The read stopped for a reason outside the named refusals, such as a network problem or a response Oda could not use. Nothing came back; try again, or point Oda at a different address.

What does "web fetch aborted" mean?

The read was stopped before it finished, for example because the step that asked for it was cancelled. Ask Oda for the page again if you still need it.

Are the limits the same in every deployment?

No. Your deployment can change the size caps, the time backstop and the redirect bound; the 2,048-character address limit is fixed. A redirect bound of zero follows no redirects at all.