The upload pipeline

This page follows an upload through the pipeline behind the DataMind Oda web interface: the streaming intake that receives it, the upload service that stores it, and the store of staged receipts that holds it until your message. For the upload as you meet it in the composer, see File uploads in the web app; for the part after the upload, see Upload receipts.

The streaming intake

• One route, raw bytes, POST only. An upload arrives as a single body of raw file bytes sent with POST; it carries no fields and no form. A request in another method is answered 405, one whose body is not raw bytes is answered 415, and one that names no chat is answered 400, each before any file byte is read. The exact texts of these refusals are in Connection and upload messages.
• The bytes stream in. The intake does not collect the file first: it passes the bytes to the upload service as chunks while they arrive, without aggregating the upload. A streamed upload is not counted against the interface's size limit for buffered requests; that limit and its default are in Connection settings and defaults.
• The address carries the file's name and its chat. The name you gave the file is not in the body: it travels as a query value of the upload's address, and so does the chat the upload belongs to. The body carries only the file's bytes, and the stored name is a cleaned version of the name you gave, as File uploads in the web app describes.
• A small answer, never cached. A finished upload answers with 200 and a short JSON result carrying the staged receipt, marked no-store so nothing keeps a copy of the answer.

The upload service

• It stores the file as it arrives. The service takes the streamed chunks and stores the file byte for byte while it receives them. A second way in exists for callers that already hold a file as bytes: the upload can arrive through a call, base64-encoded, instead of a stream; Remote calls, streams and events covers the call layer.
• The receiving chat is resolved first. Every upload names its chat, and the service resolves it before anything is stored. A chat that is not currently open is resolved through its agent, so an upload can reach a chat that is not running at the moment; an upload for a chat that cannot be resolved fails instead, with the message in Connection and upload messages.
• Failures are sorted into two kinds. A failure the service understands, such as an uploaded file that cannot be stored or a chat that ended while its upload was still in flight, keeps its own message. Anything else is reported as an internal failure whose text starts with failed to store file upload:, followed by the detail. Connection and upload messages lists what each message means and what to do.

Staged receipts

• A receipt is what the upload leaves behind. A stored upload is not attached to anything yet: the service mints a receipt carrying its own id and the stored file's details (the name, the size, and the stored reference), and stages it for the exact chat the upload named.
• The store is keyed by the chat. Staged receipts are kept with the chat itself, not in one shared list: a receipt is found only through its own chat, the store never keeps a chat alive on its own, and a chat that is gone takes its staged receipts with it.
• Binding to a message is all or nothing. When you send a message with staged files, their receipts are bound to it in one step. The binding is kept only if the message is delivered; if delivery does not complete, the binding is rolled back rather than left half-made.

The full life of a receipt, from staging to retirement, is in Upload receipts.

Questions, answered

What are the stages an upload goes through?

Three. A streaming intake receives the bytes, checking up front that the upload arrives as raw bytes and names its chat; the upload service stores the file and sorts any failure; and the result is staged as a receipt for that chat until your message carries it. The journey as you meet it is in File uploads in the web app.

Why is an upload not counted against the request size limit?

Because the intake streams the file instead of reading it whole first. A streamed upload is not counted against the size limit for buffered requests; that limit and its default are in Connection settings and defaults.

Where does the file's name come from?

From the upload's address, as a query value sent with it, not from the body: the body carries only the file's bytes. The stored name is a cleaned version of the name you gave, described in File uploads in the web app.

Does the chat have to be open when I upload?

No. If the chat is not currently open, the pipeline resolves it through its agent before storing the file, so an upload can reach a chat that is not running at the moment. An upload for a chat that cannot be resolved fails instead, with the message in Connection and upload messages.

What happens if Oda cannot store the file?

The upload fails instead of staging a receipt. A failure the service understands comes back with its own message, and anything else comes back as an internal failure starting with failed to store file upload:. What each message means, and what to do, is in Connection and upload messages.

Can a file arrive by another path?

Yes. A caller that already holds a file as bytes can send it through a call, base64-encoded, instead of streaming it; Remote calls, streams and events describes the call layer.

What does the upload get back when it finishes?

A short JSON answer with status 200, holding the staged receipt, marked no-store so it is not cached. The receipt is what binds the file to your message when you send it; Upload receipts follows it from there.

Where is a staged upload kept until I send it?

With the chat itself. Staged receipts are held per chat, a receipt is found only through its own chat, and they go with the chat when it is gone. What retires a receipt is in Upload receipts.