Every data point, three times

An import clearance starts with an email and several attached documents (PDFs, docx, xlsx, etc.). Inside come invoices, packing lists, bills of lading (B/L), health certificates and, if the goods are fish, catch certificates with their transport appendices. Sometimes they arrive as separate files, and other times all glued together in the same PDF.

With those papers you have to validate the file against regulations, prepare the customs declaration (DUA) and submit the catch certificates in TRACES, the European Commission’s system. Together with a leading customs agency we built the platform that does this work: a robot that automates the intake of documents, the extraction and structuring of their data, their validation against legal regulations, and the generation and submission to the Spanish and European customs administration systems. This article tells the hardest part: reading the data. We explain the architecture we have today, the models we used and discarded, the measurements that changed our minds and the errors we found along the way. Everything comes from the project’s history.

01The problem: one digit is expensive

In a customs document there are no small errors:

• A net weight with one digit changed throws off the declaration and the health controls.
• A container number with one letter misread breaks the cross-check with the shipping line.
• A date read as month/day instead of day/month puts a signature after the vessel’s departure, and that invalidates a certificate.

And real documents don’t help. These are cases we have found in real files:

• Scans at 150 dpi, crooked, grainy, with a stamp over the figure that matters.
• Pages sideways or upside down inside an otherwise correct PDF.
• Mixed documents: a 41-page PDF with 20 documents inside. A zoosanitary certificate appears twice, once as an unsigned draft and once signed.
• Hybrid documents: PDFs with a text layer where a fragment, sometimes less than 10 % of the page, is rasterized and is precisely the key data.
• Tables that continue on another page without a header, and 59-row packing lists spread over five sheets.
• Crossed number conventions on the same page: 21995.050 with no separator, 39,643.12 with a thousands comma and 50,000 CBM with the comma as decimal.
• Printed noise: certificates that print 3321.3900000000003 because the system that generated them does not round floating point.

The risk that worries us most is not failing. What worries us most is failing without anyone finding out.

The dangerous thing is an extraction that looks complete, with every field filled in, and carries a wrong number inside.

02First architecture: one multimodal model does everything

The initial design was the one that now seems obvious. An LLM with vision receives the page and directly returns the JSON with the document’s fields, and another LLM validates the file against the procedure’s rules. The project’s decision document justified it this way against Tesseract and against Google Vision or AWS Textract:

The stack chosen initially, which still holds: Python 3.11, Apache Airflow for the pipelines, FastAPI, PostgreSQL with JSONB, React on the front end and models served by Ollama. Soon we learned three things.

Models change underneath you

None of the models we started with is still doing the extraction. An architecture that trusts “the model” is at the mercy of the next change. The conclusion was that the model had to be a replaceable, measurable piece, not the center of the system.

The model does not warn you when it is wrong

With the first real files, the first divergences between documents of the same file appeared. They were almost never real discrepancies: they were bad readings at 150 dpi, such as a lot 31690-1 read as 31590-1 or two digits transposed. The model returned each value with the same confidence. We responded with deterministic modules:

• the check of the ISO 6346 check digit of containers, first;
• the characterization of single-character divergences, by documented confusion or by transposition of adjacent characters, afterwards.

Give the validator facts, don’t ask it for judgment

Afterwards we audited real validations:

• The question “does this mandatory document appear among those provided?” is arithmetic, and the validator failed in half the cases even though the document really was missing.
• An inspector’s signature later than the departure date, which is an objective fact in the data, was only detected in 1 out of every 4 validations.

Since then, everything that can be checked with code is checked with code and enters the validator’s prompt as a fact already verified. The model keeps the judgment, not the arithmetic.

03The turn: digitize in phases and record every decision

Fixing case by case on top of a single-call extraction had a limit: we did not know why each value came out. So we started a new pipeline, first in read-only mode on real files, with three principles:

1. Every decision is recorded. Every decision, whether by the model or deterministic, is written to a log with its phase, the result, the reason, the evidence, the discarded alternatives and the source: regla (rule), medida (measurement), modelo (model) or comparacion (comparison). There are no silent discards.
2. Never guess silently. If there is no single defensible value, the field stays empty with the candidates and the reason, and goes to human review.
3. The verifiable outranks the probable. A check digit, the PDF’s digital layer or a sum that adds up weigh more than a model’s consensus.

And a method rule, written in the pipeline’s own documentation: cost and time are not the constraint; the goal is the minimum of reading errors.

It went into production after a testing period in pre-production, with database migrations rehearsed beforehand on a copy of production.

04Format and image: measure before touching

The real type comes from the file signature, not from the extension. There are PDFs that are renamed JPEGs and “Excel” files that are HTML. Office files are read directly from their XML or cells, without OCR.

For each PDF page we measure whether it has a text layer, what images it carries and at what native resolution, the declared /Rotate and the text orientation. Then it is rendered and enhancements conditioned on what was measured are applied, recording the before and after:

• Brightness: the paper white is below 200.
• Contrast: the range between the 2nd and 98th percentiles is less than 80.
• Sharpness: the sharpness index drops below 0.16.

A page of pure vector text is not touched.

Rotation: why we ask four times

The first idea was to ask the model, with one image, how many degrees it had to be rotated. We measured it with about 700 calls on real and synthetic pages:

With a single image, on sideways pages the model got the direction right only 29 % of the time, and the closed loop did not converge in 13 % of cases. Moreover, the confidence the model returned was always 1.0, whether it was right or wrong.

A real 24-page PDF added a nuance: the vote failed on 3 of 17 digital pages of dense text with a watermark. The final rule was this:

• Pages with legible horizontal text in their digital layer are not voted.
• The rest are voted three times and rotated only with unanimity. Without it, the page stays as it is and goes to review.

05OCR and merge: transcribe faithfully and don’t trust the order

OCR is one call per page to the vision model. It is asked for a literal line-by-line transcription, without normalizing numbers or dates, and to mark [ilegible] on the specific line instead of giving up on the whole page.

Repetition loops

In a real 55-page file, 3 pages returned broken JSON. They were not long answers: they were loops, the model repeating lines of a table of empty boxes until it exhausted its token budget. The investigation gave three lessons:

• temperature=0 makes loops worse. A study we consulted measures 48 times more loops than with temperature 1.0. We use 0.1 with a repetition penalty and a wide window.
• An identical text prompt on every call can collide with the server cache. Each call now carries a random reference.
• The underlying fix was in the prompt, not the parameters: explicitly ask it not to list empty boxes and never to repeat a line already written.

With reasoning models the modern version of the problem appeared: deepseek and minimax think by default and ate the output budget before transcribing anything. They have to be called with think: false for OCR.

Merging with the digital layer without duplicating

When the page has a text layer, each OCR line is anchored to the digital line with which it shares the most “strong tokens”: words of four letters or more, or numbers compared without separators. That way we do not depend on reading order, which in a two-column form almost never matches. Each line ends up cubierta (covered), in conflicto_cifra (figure conflict; both versions are kept) or huerfana (orphan), which is content the digital layer does not have, such as a stamp.

A page is hybrid if it has 40 or more orphan characters, or 2 or more orphan figures. The threshold is absolute on purpose: a four-word stamp with the key data gets diluted in any page percentage.

06Classify and split: numbering rules

Pages are classified by text in windows of 6 pages with a step of 3, so that almost every page is judged twice. Disagreement between those two readings is a free quality signal. The cut between documents is decided with literal fields, in this order of authority:

1. Numbering: two pages with consistent numbering are never separated even if the model assigns them different types. The label is assumed to be the mistake.
2. A different identifier, or a declared document start.
3. The model saying “it is not a continuation,” as a last resort.

Each document comes out with a document status (provisional, copy, final, void or unmarked), taken from the stamped marks. A mark only counts if it occupies its line by itself, so as not to confuse the word “Stamp” in an empty box with a real stamp. In a 41-page, 20-document file it detected the 6 marked pages without any false positive.

The split generates one PDF per document by copying the original pages without re-rendering them, so the text layer reaches the following phases intact.

07Extraction: three blind readings and one decision per field

This is the core, and the design changed three times: one reading; two readings with a third one “aimed” at the field in dispute; and, finally, three independent, blind readings, in which none sees what the others answered and each field is decided by majority. All three are always stored. If all three differ, the field goes to review with the three in view.

Two readings that agree are not two readings that are right.

That is why we discarded reading list fields page by page, which made readings converge on the same wrong value, and why the majority is not the last word.

Numbers are decided in code, not in the model

For numeric fields the model is asked for the literal as printed. If you ask it for the number already “decided,” it normalizes, and it does so even where the real convention was another. The decision is made by a deterministic cascade that stops at the first criterion that leaves a single candidate:

The thousands and decimals convention does not belong to the document, it belongs to each field. On that basis there are checks that have already caught real errors:

• Witness balancing: the sum of a column is confirmed against the printed total.
• Dates by column: DD/MM or MM/DD is decided for the whole column, never cell by cell.
• Packages read as kilos: a packing list whose breakdown was 1,998 cartons had been read as kilos. If the sum of weights per line matches the total number of packages and diverges from the file’s known weights, those weights are voided.

A case that taught us to measure with the full pipeline

A scanned 7-page certificate came out with 15 items when there were 9, and all three readings brought the 15. The cause was in our chunking:

• Images are sent in batches of at most 4 images or 4 MB, and the OCR text of all the pages traveled attached only to the first one.
• The first batch already saw the 9 rows through the text, and the second one read again those from its pages from the image.
• The telltale sign: rows that came from OCR had a decimal comma and those that came from the image had a decimal point.

The first fix corrected that certificate, but it made a 59-row packing list lose the 11 rows of its last page, a continuation without a header, about 30 % of the time. With a notice in the batch text (“also extract the rows of the pages not shown as an image”) it failed 0 of 28 times (Fisher’s exact test, p ≈ 0.002). We added an arbiter: if the readings differ in the number of rows, the one that matches the printed total wins. “The longest wins” was discarded because in the history, over 1,192 tables, the longest was not right even once in the disagreements.

The digital layer prevails

A value that appears literally in the PDF’s text layer, or in an Excel’s cells, has not gone through optical recognition and cannot have a changed character. Each value carries its provenance (digital, OCR or conflict) and, when two documents contradict each other, the digital one rules without a vote. Provenance is checked by value, not by page, because a hybrid page has a digital part and a scanned part.

Later we extended it because of a real case: a guard did not recognize a packing list titled “Packing certificate” and the three readings came out empty, while the digital layer carried all the data. Now the digital layer prevails also when the model returns nothing.

08Zones: knowing which box each figure comes from

A per-page OCR returns a flat list of lines, and in a two-column form adjacent labels fuse together:

I.24. Total number of packages I.25. Total quantity
I.26. Total net weight I.26. Total gross weight
1320 Cardboard box
22440 kg 23760 kg

The extractor reconstructed from context which was net and which was gross. It got it right, but it was not auditable. The zones phase splits the page before reading, with three detectors in order:

1. Text-layer blocks, exact and free where there is one.
2. OpenCV morphology on the rotated image. On a real certificate it extracts its 39 boxes, and on a packing list with no lines it invents none.
3. The model pointing at the table over a reference grid, accepted only if three different grids agree.

09Consistency: the file as a witness

When the three readings give no majority, the rest of the file usually has the answer. From a catalog of 41 candidate cross-checks, consistency applies deterministic deciders over the set of candidates. It never invents a value that no reading gave:

• ISO 6346 check digit of the container.
• Nomenclature code against the declared species.
• Rigid-format identifiers that validate themselves, such as REX codes or the border inspection post code.
• Majority among different documents, not among readings of the same one. With two declarants a disagreement never has a majority, so we look for new declarants, such as the B/L hidden in a box of the health certificate.

In the demo we recorded with fictitious documents, the scanned appendix read the container as CCVU4182735. It fails the check digit, and four other documents declare OCVU4182735, which differs by a single letter. The platform corrects it and leaves the field flagged for review.

Applying corrections without losing information

1. Explicit catalog of decisions that may correct. Everything else touches nothing.
2. Additive corrections: the original reading is never overwritten, so it is always possible to say what the extraction read and what consistency applied.
3. Compare-and-swap: a correction is only made where the value is still the one the cross-check examined. It protects cells with two codes, such as a legitimate seal next to a bad reading.
4. Applying is not closing the review. The review is only closed when the code verifies the value. A correction by corroboration or by a model’s judgment is applied with reservations and the field stays in the queue.

10An external arbiter as a fallback: a specialist third-party service

Fields that are still pending review when consistency ends are sent to the API of a service specialized in data extraction and structuring with the schema of the document type itself: a single call parses and extracts. That cross-check exposed a flaw in our schema. It said “8 digits” for the nomenclature code, and faced with a certificate that prints it with 6 (030743), any extractor inflated it to 03074300 to comply with the instruction.

The arbiter makes one call per page, cached by hash, and only replaces the fields that were under review. On the same page and with the same schema, it returned a correct description in 3 of 5 calls and mixed up the column in the other 2. Overwriting everything would have imported that noise.

Making the sums add up, with no tolerance

Invoices and packing lists are checked by their own arithmetic. If the readings diverge or do not add up, they are cross-checked with the fallback: if its result adds up it is recorded, and if not it stays in review. The tolerance is that of the printed rounding and nothing more. With the 0.5 % we used before, an error of 5,000 out of 5.8 million got through.

The case that made us change was a scan in which all three readings were identical and carried an extra digit in the total.

Neither majority nor unanimity guarantees anything; arithmetic does.

11Which model, and how we chose it

Later we compared three candidates with the full pipeline on a file of 7 documents and 79 pages, using the fallback as arbiter (it had got 24 of 24 hand-verified fields right):

gemma4 misread the year of dates in 18 or 19 of 20 cases, and between two of its passes it only agreed on 86 % of fields. A second cross-check, with 40 files from 5 issuers, forced us to add nuance:

• In precision there was a tie, around 88–90 %.
• Where there was stable disagreement, deepseek was right 53 times and gemma4, 20.
• deepseek leaves 18 % more values unfilled, which for us is better than filling them wrong.

Today extraction uses deepseek-v4.1-flash with think: false. Before adopting it we had to fix our parser, which discarded its answers when it returned a single object instead of a list. Before blaming the model, it pays to read what it answers.

We also tested an external decider model, pplx-decider: it got 24 of 24 orientations and 22 of 24 classifications right, with one error given with confidence 0.93. We gave it no authority. A decider that is sure of itself and wrong is exactly what we are trying to avoid.

12The method

1. Always with real files, processed end to end in pre-production. Synthetic figures only give direction.
2. Diagnose against the paper: before fixing, re-read the raw literal from the log and, if needed, re-render the page at the same DPI the model sees.
3. Measure with the full pipeline and with several document types. A correct fix for one document can break another.
4. Independent design review: when in real doubt we consult a second model, always a Frontier MoE, and verify its proposal against the code and the data before applying it.
5. Every commit tells the real case that motivated it. That way, more than a thousand commits read like a lab notebook.

13What is still unresolved

• Coverage of hybrid documents by regions: corroborating by the real position of each image on the page, not just by line.
• More cases with known ground truth: files with every field verified by hand, which are what allow us to truly measure each change.
• Zones as the source of each value in all document types, not only in box-based forms.

Conclusions

Vision models are excellent at reading and bad at warning that they have made a mistake. The system around them has to do that second part: measure before transforming, read more than once without contaminating the readings, prefer the verifiable to the probable, use the whole file as a witness and leave every decision written down. What it cannot resolve with certainty, it puts in front of a person with the evidence beside it.

There is a significant technological distance between the reading of an arbitrary document by an AI application (such as ChatGPT, Claude or others) and a Data Extraction Agent robot that automates the industrial process of thousands of heterogeneous documents, and guarantees reliability, traceability, security and scalability.