Anonymous web fetch safety

DataMind Oda can read a public page without going through your company's web access. That read uses the anonymous path: the request is made from this computer, carries no credentials, and only public addresses are ever contacted. This page covers the path's safety rules and the refusals you may see; the calls themselves are on Searching and reading the web, and which path answers a read is on Web search providers.

Which addresses it opens

A read starts with an address check, and the check is strict:

• http and https only. Any other scheme is refused: unsupported URL scheme "<scheme>" (only http and https are allowed).
• No credentials inside the address. An address that carries a user name or a password is refused: credentials in URLs are not allowed.
• At most 2,048 characters. The address limit is fixed. A longer address is refused: URL exceeds the maximum length of 2048.

A refused address means the read never starts; ask Oda to open a plain http:// or https:// address instead.

Public pages only

The host name is resolved once, and every address it returns is checked before any connection is made; only public addresses are allowed through.

• One non-public answer refuses the read. When any address in the answer is private, loopback or otherwise non-public, the whole read is refused: URL hostname "<h>" resolves to a non-public IP address. The check is all-or-nothing, so one non-public address among public ones is enough.
• The checked addresses are the ones used. The read connects to the addresses that passed the check, and the name is not resolved again for it, so nothing can point the connection at a non-public address after the check.
• Translations count too. An address translation that would reach a non-public address is refused: URL hostname "<h>" resolves through NAT64 to a non-public IPv4 address.
• A name with nothing to reach. A name that resolves to no addresses is refused: hostname "<h>" resolved to no addresses. So is a name that resolves to an invalid address: hostname "<h>" resolved to an invalid IP address.

These checks keep the path away from private and internal services: a page can be read only where it is genuinely on the public web.

Redirects stay on the same site

A page can redirect a read to another address; redirects are followed only within the same site, meaning the same scheme, host name and port.

• Same site only. A redirect to another origin stops the read: cross-origin redirect to <origin> is not followed automatically; retry against that URL directly. Ask Oda to read the address named in the message directly.
• Every hop is re-checked. A redirect target goes through the same checks as a direct read, including the public-address check.
• A redirect budget. Up to five redirects are followed by default; the budget can be set to zero, which follows no redirects. Past the budget the read stops: exceeded the maximum of <n> redirects.
• A redirect with no destination. A redirect response that names no new address fails the read: redirect response (HTTP <status>) without a Location header.
• A key never travels through a redirect. When a read carries a key, a redirect response ends the read before the new address is contacted, so the key cannot be sent on.

Size, type and time limits

Every read runs under limits; these are the defaults:

• Readable text only. An HTML page is read and converted to text; plain text, JSON and XML pages are read as they are. Any other type, such as an image or a PDF, is refused: unsupported content type "<ct>".
• A response known to be too large is refused. When the size is known before the body is read and it is over the cap, the read stops with response exceeds the maximum of <n> bytes. A response that grows past the cap while it is being read is stopped at the cap instead.
• The page text kept is capped. The reader keeps at most 100,000 characters of a page's text by default.
• A slow read is stopped. The anonymous reader keeps a 30-second backstop of its own; a read that passes it reports web fetch timed out. Timeouts and time zones covers how that backstop sits behind the tool's own budget.

Reads that start without a prompt

• No approval prompt. A public read does not stop for a confirmation: in the presets that ship with Oda, the page reader is available in every access mode, and a read starts without asking you first.
• If you want each read confirmed. Your deployment can require approval for reads, or the web tools can be switched off; Create a custom preset shows the Web search and pages switch that turns them off for a chat.

Questions, answered

Can Oda read a page on our internal network?

No. Only public addresses are contacted. When the host name answers with any non-public address, the whole read is refused with URL hostname "<h>" resolves to a non-public IP address; the check is all-or-nothing, so one non-public answer among public ones already refuses the read.

Can Oda read an address that is not http or https?

No. Only http and https addresses are opened; any other scheme is refused with unsupported URL scheme "<scheme>" (only http and https are allowed).

Why was an address refused for carrying credentials?

An address that carries a user name or a password inside it is refused with credentials in URLs are not allowed. Ask Oda to read the plain address instead.

The page redirected Oda to another site. Why did the read stop?

Redirects are followed only within the same site, meaning the same scheme, host name and port. A redirect to another origin is refused with cross-origin redirect to <origin> is not followed automatically; retry against that URL directly; ask Oda to read the address named in the message directly.

How many redirects can one read follow?

Five by default. The budget can be set to zero, which follows no redirects. A chain past the budget stops with exceeded the maximum of <n> redirects.

Can Oda read a PDF or an image?

No. Only readable text comes back through this path: HTML is converted to text, and plain text, JSON and XML are read as they are. Any other type is refused with unsupported content type "<ct>". Open the file yourself, or give Oda a text page to read.

What does response exceeds the maximum of <n> bytes mean?

The response was larger than the size cap, 5,000,000 bytes by default, and its size was known before the body was read, so the read was refused. A response that grows past the cap while it is being read is stopped at the cap instead.

Is anything about me sent when this path reads a page?

No. A read through this path carries no credentials of yours, so the page cannot tie the request to your account or your company.

What does hostname "<h>" resolved to no addresses mean?

The host name did not resolve to any address, so there was nothing to connect to and the read was refused. Check the address for a typo, then try again.