Anonymous web fetch safety

DataMind Oda can read a public page without going through your company's web access. That read uses the anonymous path: the request is made from this computer, carries no credentials, and only public addresses are ever contacted. This page covers the path's safety rules and the refusals you may see; the calls themselves are on Web search and page fetch, and which path answers a read is on Web search providers.

What the built-in reader is

• A reader on this computer. It opens the address and returns the page's text. The read does not go through your company's DataMind server, and it needs no sign-in.
• Anonymous by design. It sends no credentials of any kind, so only pages that are open to anyone can be read. A page that needs a sign-in cannot be read this way.
• It announces itself. A direct read says it is DataMind Oda, and it never pretends to be an ordinary browser, so a site can tell that Oda is reading the page.
• Chosen by your deployment. When your deployment uses it for page reading, every direct read follows the rules on this page.

Which addresses it opens

A read starts with an address check, and the check is strict:

• http and https only. Any other scheme is refused: unsupported URL scheme "<scheme>" (only http and https are allowed).
• No credentials inside the address. An address that carries a user name or a password is refused: credentials in URLs are not allowed.
• At most 2,048 characters. The address limit is fixed. A longer address is refused: URL exceeds the maximum length of 2048.
• An address that parses. Text that does not parse as an address is refused before anything is fetched: invalid URL: <input>.

A refused address means the read never starts; ask Oda to open a plain http:// or https:// address instead.

Public pages only

The host name is resolved once, and every address it returns is checked before any connection is made; only public addresses are allowed through.

• One non-public answer refuses the read. When any address in the answer is private, loopback or otherwise non-public, the whole read is refused: URL hostname "<h>" resolves to a non-public IP address. The check is all-or-nothing, so one non-public address among public ones is enough.
• The checked addresses are the ones used. The read connects to the addresses that passed the check, and the name is not resolved again for it, so nothing can point the connection at a non-public address after the check.
• Translations count too. An address translation that would reach a non-public address is refused: URL hostname "<h>" resolves through NAT64 to a non-public IPv4 address.
• A name with nothing to reach. A name that resolves to no addresses is refused: hostname "<h>" resolved to no addresses. So is a name that resolves to an invalid address: hostname "<h>" resolved to an invalid IP address.
• Behind a proxy. When Oda runs behind a proxy, a public read goes through it. A private or loopback address written as numbers is still refused and never handed to the proxy; Proxy support covers the proxy settings themselves.

These checks keep the path away from private and internal services: a page can be read only where it is genuinely on the public web.

Redirects stay on the same site

A page can redirect a read to another address; redirects (the 301, 302, 303, 307 and 308 responses) are followed only within the same site, meaning the same scheme, host name and port.

• Same site only. A redirect to another origin stops the read, and its target is never contacted: cross-origin redirect to <origin> is not followed automatically; retry against that URL directly. Ask Oda to read the address named in the message directly.
• Every hop is re-checked. A redirect target goes through the same checks as a direct read, on scheme, credentials, length and a public destination.
• A redirect budget. Up to five redirects are followed by default; the budget can be set to zero, which follows no redirects. Past the budget the read stops: exceeded the maximum of <n> redirects.
• A redirect with no destination. A redirect response that names no new address fails the read: redirect response (HTTP <status>) without a Location header.
• A key never travels through a redirect. When a read carries a key, a redirect response ends the read before the new address is contacted, so the key cannot be sent on.

Size, type and time limits

Every read runs under limits; these are the defaults:

• Readable text only. An HTML page is read and converted to text; plain text, JSON and XML pages are read as they are. Any other type, such as an image or a PDF, is refused, and so is a page that names no content type: unsupported content type "<ct>".
• Declared character set. A page is decoded using the character set the response declares, and a set the reader does not recognise is refused rather than guessed at.
• A response known to be too large is refused. When the size is known before the body is read and it is over the cap, the read stops with response exceeds the maximum of <n> bytes. A response that grows past the cap while it is being read is cut at the cap instead, and the result is marked as truncated; a body that arrives exactly at the cap is not marked.
• The page text kept is capped. The reader keeps at most 100,000 characters of a page's text by default; longer text is cut and marked as truncated.
• A slow read is stopped. The anonymous reader keeps a 30-second backstop of its own; a read that passes it reports web fetch timed out. The backstop caps the resource, not the call: a read asked for from a chat can also be stopped earlier by the call's own time budget. Timeouts and time zones covers both layers.
• Deployment adjustments. Your deployment can change the size caps, the time backstop and the redirect bound; the 2,048-character address limit is fixed. Each value is checked when it is set: the caps and the backstop must be positive, finite numbers, the backstop cannot exceed 2,147,483,647 milliseconds, and the redirect bound is a whole number, where zero follows no redirects.

When a read fails for another reason

• web fetch failed: <error>: the read stopped for a reason outside the named refusals, such as a network problem or a response Oda could not use. Try again, or point Oda at a different address.
• web fetch aborted: the read was stopped before it finished, for example because the step that asked for it was cancelled. Ask again if you still need the page.

Reads that start without a prompt

• No approval prompt. A public read does not stop for a confirmation: in the presets that ship with Oda, the page reader is available in every access mode, and a read starts without asking you first.
• If you want each read confirmed. Your deployment can require approval for reads, or the web tools can be switched off; Create and edit agent presets shows the Web search and pages switch that turns them off for a chat.

Questions, answered

Can Oda read a page on our internal network?

No. Only public addresses are contacted. When the host name answers with any non-public address, the whole read is refused with URL hostname "<h>" resolves to a non-public IP address; the check is all-or-nothing, so one non-public answer among public ones already refuses the read.

Why was my page address refused?

The address must be an http or https address, must not carry a user name or password, and must stay within 2,048 characters. An address that breaks one of these rules, or that does not parse as an address at all, is refused with its own message, such as unsupported URL scheme "<scheme>" (only http and https are allowed) or credentials in URLs are not allowed, before anything is fetched. Ask Oda to read the plain address instead.

The page redirected Oda to another site. Why did the read stop?

Redirects are followed only within the same site, meaning the same scheme, host name and port. A redirect to another origin is refused with cross-origin redirect to <origin> is not followed automatically; retry against that URL directly, and the other site is never contacted; ask Oda to read the address named in the message directly.

How many redirects can one read follow?

Five by default. Your deployment can set a different budget, and zero follows no redirects. A chain past the budget stops with exceeded the maximum of <n> redirects.

What if a redirect response has no destination?

The read fails before anything further is fetched: redirect response (HTTP <status>) without a Location header. Nothing was read from that address. Try again, or give Oda the page's final address if you know it.

Can Oda read a PDF or an image?

No. Only readable text comes back through this path: HTML is converted to text, and plain text, JSON and XML are read as they are. Any other type is refused with unsupported content type "<ct>". Open the file yourself, or give Oda a text page to read.

What does response exceeds the maximum of <n> bytes mean?

The response was larger than the size cap, 5,000,000 bytes by default, and its size was known before the body was read, so the read was refused. A response that grows past the cap while it is being read is cut at the cap instead and marked as truncated.

Is anything about me sent when this path reads a page?

No. A read through this path carries no credentials of yours, no sign-in and no key, so the page cannot tie the request to your account or your company. It does say it is DataMind Oda, so the site can tell that Oda is reading. A page that requires a sign-in cannot be read this way.

What does hostname "<h>" resolved to no addresses mean?

The host name did not resolve to any address, so there was nothing to connect to and the read was refused. Check the address for a typo, then try again.

What does the NAT64 address message mean?

Some networks translate a name from one address family to another. When that translation lands on a private address, Oda refuses the read like any other non-public destination: URL hostname "<h>" resolves through NAT64 to a non-public IPv4 address. There is nothing to correct on your side; the destination stays out of reach for a direct read.

Is a private address allowed when Oda runs behind a proxy?

No. A private or loopback address written as numbers is refused even on the proxy path: it is never handed to the proxy. The same addresses are refused when no proxy is set.

What do web fetch timed out, web fetch failed and web fetch aborted mean?

web fetch timed out means the read passed the reader's 30-second backstop. web fetch failed: <error> means the read stopped for another reason, such as a network problem. web fetch aborted means the read was stopped before it finished, for example because its step was cancelled. Try again, or point Oda at a lighter page or a different address.