Direct web fetching

When Oda reads a page for a task, the read goes through one of the readers set up for your deployment. The built-in one opens public pages directly from this computer and sends no credentials; this page covers its rules: the addresses it can open, the destinations it refuses, the redirects it follows and the limits it keeps. The tool side and the chat cards are covered on Web search and page reading, and how Oda chooses the reader is on Web search providers.

What the built-in reader is

• A reader on this computer. It fetches the address and returns the page's text. The read does not go through your company's DataMind server, and it needs no sign-in.
• Anonymous by design. It sends no credentials of any kind, so only pages that are open to anyone can be read; the questions below cover what that rules out.
• Chosen by your deployment. When it is the reader for page reading, every direct read follows the rules on this page.

Which addresses it can open

• Web addresses only. An address with another scheme is refused: unsupported URL scheme "<scheme>" (only http and https are allowed).
• No credentials inside the address. A username or password in the address is refused: credentials in URLs are not allowed.
• At most 2,048 characters. A longer address is refused: URL exceeds the maximum length of 2048.
• An address that parses. Text that does not parse as an address is refused: invalid URL: <input>.

Which destinations it will not reach

• Public internet addresses only. Before connecting, Oda resolves the hostname once and checks every address it came back with; if any of them is not a public internet address, the whole read is refused: URL hostname "<hostname>" resolves to a non-public IP address.
• That covers private places. Addresses on your company network, your home network or this computer itself are out of reach. A loopback or private address written as numbers is refused the same way, even when Oda runs behind a proxy.
• The check holds for the whole read. The connection is opened only to the addresses that passed the check, so the destination cannot change afterwards.
• Address translations are checked too. A hostname that resolves through an IPv6-to-IPv4 translation to a private address is refused like any other non-public destination.
• Unresolvable names. A name that resolves to nothing, or to something that is not a usable address, is refused: hostname "<hostname>" resolved to no addresses, hostname "<hostname>" resolved to an invalid IP address.

Redirects

• Within the same site. A redirect that stays on the same scheme, host and port is followed. A redirect that points to another site is refused and its target is never contacted: cross-origin redirect to <origin> is not followed automatically; retry against that URL directly. Give Oda the final address directly instead.
• Up to five by default. A chain longer than the bound is refused: exceeded the maximum of <n> redirects. Your deployment can set a different bound; zero follows no redirects at all.
• Reads that carry credentials are different. They never follow a redirect at all, and the redirect's target is never contacted; Web search providers covers that side.

Size, type and time limits

• Response size. A response that announces more than 5,000,000 bytes is refused before it is downloaded: response exceeds the maximum of <n> bytes. A body that grows past that size while it arrives is cut short instead and the result is marked as truncated; a response that arrives exactly at the limit is not marked.
• Page text. Up to 100,000 characters of page text come back; longer text is cut and marked. The chat's card shows the status and any cut; Web results in chat covers the cards, and text too large for the chat is stored in full on this computer (Large results and spill files).
• Text-carrying pages only. HTML, plain text, JSON and XML can be read. Anything else is refused, including a page that names no content type: unsupported content type "<type>".
• Declared character set. The page's character set is taken only from what the response declares; a set the reader does not recognise is refused rather than guessed at.
• A time budget. A read has a budget of 30 seconds by default and is stopped when it runs longer: web fetch timed out. A read asked for from a chat can also be stopped earlier by the call's own time budget; Timeouts in Oda covers both layers.

Questions, answered

Why can't the built-in reader open a page on my company network?

Because only public internet addresses are allowed. Oda resolves the address before connecting and refuses the read when any answer is not a public address, so internal services and other computers on your network are out of reach by design. The message reads URL hostname "<hostname>" resolves to a non-public IP address.

Why was my page address refused?

The address must be a web address (http or https), must not carry a username or password, and must stay within 2,048 characters. An address that breaks one of these rules, or that does not parse as an address at all, is refused with its own message before anything is fetched.

Why did my read stop with a cross-origin redirect?

The page redirected to another site. The reader follows redirects only within the same site and never contacts the other site's address, so nothing was read from it. The message is cross-origin redirect to <origin> is not followed automatically; retry against that URL directly; ask Oda again with that final address.

How many redirects does Oda follow?

Up to five by default, all within the same site; your deployment can set a different bound, and zero follows no redirects at all. A chain longer than the bound is refused with exceeded the maximum of <n> redirects.

Why was my page refused for its size?

The response announced more than the reader allows (5,000,000 bytes by default), so it was refused before the download: response exceeds the maximum of <n> bytes. A smaller page, or a more specific address on the same site, can be read instead.

Why can't the reader read a PDF or an image?

Only text-carrying pages are read: HTML, plain text, JSON and XML. A PDF, an image or another file kind is refused with unsupported content type "<type>"; a page that names no content type counts as unsupported too.

Why did the read time out?

The read did not finish within its time budget and was stopped: web fetch timed out. The budget is 30 seconds by default, and a read asked for from a chat can also be stopped earlier by the call's own time budget. Try again, or point Oda at a lighter page.

Can the reader open a page that needs a sign-in?

No. The reader sends no sign-in, no key and no other credential; its requests are anonymous. A page that requires a sign-in cannot be read through it, and that is also why only public pages can be read this way.

Does a read use my proxy settings?

When Oda runs behind a proxy, a read of a public page uses it. Addresses on your computer or your private network stay refused either way; Proxy support covers the proxy settings themselves.

Why did my address fail to resolve?

The name did not resolve to a usable address. Oda reports hostname "<hostname>" resolved to no addresses when nothing came back, and hostname "<hostname>" resolved to an invalid IP address when what came back was not a usable address. Check the address for a typo, then try again.