File conversion
DataMind turns a PDF document into text, OCR output or Markdown. This page covers the two conversion modes, the settings a document can need, and what a conversion gives back.
The conversion runs as part of other flows; for example, a PDF uploaded as a training card becomes Markdown this way before its text joins the card (Upload files as training cards).
Run a conversion
- Choose a mode: the standard mode for quick text extraction, or the layout-aware mode when the document's layout and tables matter.
- Set the settings the document needs, or keep the defaults.
- Start the conversion with the PDF, whole or as a page range.
- Read the result as a structured result, as Markdown, or as both; a long conversion can be followed while it runs, or cancelled.
Choose a conversion mode
| Mode | What it does | Use it when |
|---|
| Standard | Fast text extraction with a per-page OCR fallback for scanned pages. | The document has a readable text layer and speed matters. |
| Layout-aware | Layout and table-aware extraction; slower, higher fidelity. | Headings, lists and tables must survive into the output. |
- Both modes read the same PDF and both accept the same page range.
- The layout-aware mode keeps a document's structure: headings come back as Markdown headings, list items as bullets, and tables as Markdown tables.
- It is slower. Expect roughly two to three seconds per page on a two-core machine, and around one to two gigabytes of memory while it runs.
Settings for a conversion
- Page range: convert the whole document, or from a first page to a last page. Pages count from 1. A last page past the end is clamped to the document's last page, not refused, and the last page must be at or after the first.
- OCR: in the standard mode you can choose automatic (the default), force or off. Automatic reads a page's own text first and falls back to OCR when the page looks scanned. Force reads every page with OCR and ignores its text layer. Off never runs OCR, so a scanned page comes back with no text. The layout-aware mode reads scanned pages with OCR and has no switch.
- Text character threshold: 20 characters by default. A page whose own text is shorter than this counts as scanned, so it is read with OCR, or left empty when OCR is off.
- Image regions: on a page that has text, an image covering at least 0.05 of the page area is read with OCR as a region too, and the page comes back as a hybrid of its own text and the regions. Raise the ratio to 1 to switch region reading off.
- Tables: extraction is on by default in the standard mode. A table's area is kept out of the page's text, so its words are not repeated.
- Layout: off by default in the standard mode. Turn it on to keep the page's physical arrangement in its extracted text.
- Tolerances: the horizontal and vertical tolerances, 3.0 by default, tune how characters group into words and lines in the standard mode.
- OCR resolution: 300 DPI by default, adjustable from 72 to 600 in the standard mode and from 150 to 600 in the layout-aware mode, which detects layout at 150 DPI.
- OCR languages: Georgian and English.
- Minimum word confidence: 0 in the standard mode and 40 in the layout-aware mode. OCR words scoring below it are dropped as noise.
- OCR engine settings: the segmentation the reader assumes for a whole page and for an image region, and its engine mode, each have a default. Change them only when a particular scan reads poorly.
Read the result
- A conversion returns a structured result, Markdown, or both. The standard mode returns the structured result by default; the layout-aware mode returns Markdown by default.
- The structured result names the file, the page count, how many pages ran, how long the conversion took, the settings that were applied, and one record per page. A layout-aware result also reports any pages it had to skip.
- Each page record carries its page number, its text and any tables. A hybrid page also carries the OCR text of its image regions.
- A page's method says how it was produced: text from the page's own layer, OCR when the page was read with OCR, hybrid for a page that combines both, and none for a scanned page skipped because OCR was off.
- An OCR page carries a confidence score: the mean word confidence, from 0 to 100. Read it as a quality signal. 90 and above is a clean scan. 70 to 90 is usable, with scattered errors, so it suits retrieval and is risky for exact quotes. Below 70 is a poor scan whose text will mislead. A page with no OCR, or one where nothing was recognised, carries no score.
- Markdown output joins the pages with a horizontal rule between them and renders tables as Markdown tables. In a layout-aware result an empty page produces no entry, so a gap in the page numbers means an empty page. A page that failed is counted in the result, skipped, and does not stop the rest of the document.
Follow a conversion while it runs
- A conversion can also run as a job you follow while it works, instead of one request that returns the whole result.
- The progress stream reports pages done, pages total and a percentage as pages finish. It ends with done when the result is ready, or error or cancelled when the conversion stops.
- The result stays available for five minutes after done. Read it within that window; afterwards the job is cleared.
- A running conversion can be cancelled while it is processing. The cancellation is accepted with Cancellation requested, and the conversion stops at its next checkpoint.
- If the progress stream drops, reconnect and it resumes from the last event you saw. Do not stay away: a running conversion whose listeners have all left is cancelled about ten seconds after the last one disconnects.
- Conversions run one at a time. A conversion that arrives while another is running waits its turn.
- The conversion keeps no queue of its own. Queuing, retries, and file-size or page limits belong to the flow that asks for a conversion.
When a conversion cannot finish
| What happens | What you see |
|---|
| The file does not carry the PDF signature near its start. | File is not a PDF. The refusal comes before any work starts. |
| The PDF cannot be opened: damaged, incomplete or password-protected. | corrupt, truncated, or password-protected PDF. |
| The first page asked for is beyond the document. | The refusal names the first page and the document's page count. |
| The last page asked for is before the first. | The range is refused; the last page must be at or after the first. |
| The connection goes away while the conversion runs. | Client disconnected; the conversion stops. |
| A layout-aware conversion runs past its time budget. | The conversion stops with a timeout. |
| The result is read before the conversion is done. | Conversion is still in progress. |
| The conversion was cancelled. | Conversion was cancelled; there is no result. |
| The conversion stops with no more specific reason. | Conversion failed. |
| The job is gone: finished and cleared, or never existed. | The job is reported as not found. |
| Cancellation is asked for after the conversion already finished. | The cancellation is refused as already finished; read the result instead. |
Questions, answered
Which conversion mode should I choose?
Use the standard mode when the PDF has a readable text layer and you want a quick result. Use the layout-aware mode when headings, lists and tables must survive into the output. It is slower: roughly two to three seconds per page on a two-core machine.
Can I convert only some pages of a document?
Yes. Set a first page and a last page; pages count from 1. A last page past the end is clamped to the document's last page rather than refused, and the last page must be at or after the first. A first page beyond the document is refused, naming the page count.
What happens to a scanned page?
It is read with OCR when the OCR setting is automatic (the default) or force. Automatic falls back to OCR when a page's own text is shorter than the text character threshold. With OCR off, a scanned page comes back with no text and its method reads none.
What does the OCR confidence score mean?
It is the mean word confidence, from 0 to 100, across the OCR on that page. Treat 90 and above as a clean scan, 70 to 90 as usable but with scattered errors, and below 70 as poor: text built from it will mislead. A page with no OCR carries no score.
Why is a page number missing from a layout-aware result?
An empty page produces no entry, so a gap in the page numbers means an empty page. A page that failed is counted separately in the result and skipped, and the rest of the document still converts.
Can I get Markdown instead of a structured result?
Yes. Ask for Markdown to get one Markdown document, or for both to get the structured result with its Markdown alongside. The standard mode returns the structured result by default; the layout-aware mode returns Markdown by default.
How long after a conversion finishes can I read the result?
Five minutes after the progress stream reports done. After that the job is cleared and the result is no longer available; convert the document again if you still need it.
Can I cancel a conversion while it runs?
Yes, while it is processing. The cancellation is accepted with Cancellation requested, and the conversion stops at its next checkpoint. A conversion that already finished cannot be cancelled, and the refusal says it already finished. A cancelled conversion has no result.
My progress stream dropped. Can I resume it?
Reconnect with the stream and it resumes from the last event you saw, so no progress is lost. Do not leave it long: a running conversion whose listeners have all left is cancelled about ten seconds after the last one disconnects.
Why was my file refused?
A file that does not carry the PDF signature near its start is refused with File is not a PDF. A PDF that is damaged, incomplete or password-protected is refused as corrupt, truncated, or password-protected PDF. Export the document as a PDF again, or get a clean copy without a password, and retry.
Does the conversion change my PDF?
No. The conversion reads a copy of the file, and that copy is deleted when the conversion ends. Your document itself is never changed.
Is there a file size or page limit?
Not from the conversion itself. It keeps no queue and sets no file-size or page limit; the flow that asks for a conversion applies its own limits, so check the feature you are using.