We Ran 219 Italian Business Documents Through an Offline AI. Here Are the Numbers.

The question we set out to answer

Can a 7-billion parameter open-source model, running on a €900 GPU, extract structured data from Italian business documents accurately enough for production use? Not in a controlled demo. Not on hand-picked examples. On a proper, systematic benchmark with 219 documents, a known correct answer for every single field, automated comparison logic, and published results.

The Italian business document landscape is notoriously complex. Invoices must comply with specific fiscal formats including 11-digit VAT numbers (Partita IVA) with a proprietary checksum algorithm. Payslips carry a 16-character Codice Fiscale generated from name, birthdate, and municipality using the full ODD/EVEN table algorithm with homocody handling. Delivery notes follow the D.P.R. 14/08/1996 n. 472 format. Bank statements include IBANs, transaction codes, and running balance calculations. These aren't generic documents — they're heavily localized, and any serious extraction system needs to handle all of it reliably.

This benchmark is the first systematic scientific test of DataUnchain's processor v2.0 against this document landscape. Every number here is real, every method is fully documented, and every result is verified against ground truth using automated comparison logic.

Why a ground truth benchmark matters

Most demos of AI document systems show screenshots of successful extractions. That's not science — it's marketing. A real evaluation requires three things: a large enough corpus that statistical noise averages out; a known correct answer for every field in every document; and fully automated comparison that leaves no room for subjective interpretation of results.

The concept of ground truth is the cornerstone of machine learning evaluation. For each document in our corpus, a JSON file contains the expected values for every extractable field — the invoice number, the supplier's VAT number, the issue date, the taxable amount, the VAT, the total. After the system processes a document, we compare its output against the ground truth file automatically. A date either matches or it doesn't. An amount is either within tolerance or it isn't. There is no grey area.

We generated the corpus synthetically, meaning we built the PDFs ourselves using the fpdf2 library with realistic Italian business data. This gives us perfect ground truth from the start. The data is authentic: VAT numbers are generated with the real 11-digit checksum algorithm. Fiscal codes use the complete official algorithm including Belfiore municipality codes, the ODD/EVEN character tables, the final check digit calculation, and homocody handling for duplicate codes. Dates are in real Italian formats. Amounts are in euros with Italian thousand-separator notation (period as thousands separator, comma as decimal).

The scan simulation: testing real-world conditions

70% of the corpus — approximately 153 documents — was subjected to a controlled degradation pipeline to simulate real-world office scanning. This is not a simple JPEG save. We applied a multi-step transformation sequence: Gaussian noise at varying intensity levels, random rotations in the ±3° range, JPEG compression at quality levels between 60 and 85 (the typical output range of office network scanners), overlaid stamps and watermarks in semi-transparent layers, brightness and contrast variations, and simulation of slight paper creases and perspective distortions from non-flat scanning surfaces.

These transformations reproduce what happens in the vast majority of real document workflows: a document is printed, signed, stamped with a company seal, scanned with an inexpensive office scanner, and the resulting file is compressed before being emailed or archived. The output is a JPEG-based PDF with visible degradation artifacts. This is the reality of document processing in Italian businesses, and any system that only works well on pristine digital PDFs is not ready for production.

Ground truth verified mathematically

For documents with financial figures, the ground truth is not just a collection of correct values — it is a mathematically consistent set. For every invoice in the corpus: taxable_amount + vat_amount = total exactly to the cent. For every payslip: gross_pay - deductions = net_pay precisely. For every bank statement: opening_balance + total_credits - total_debits = closing_balance to the cent.

This mathematical consistency serves a dual purpose. First, it lets us test the system's math check feature: when the system extracts all the financial fields, it independently verifies the arithmetic. A 100% math check score means the system not only read the right numbers, but read them consistently with each other across the document. Second, it means that any inconsistency in the extracted data is a genuine error, not an artifact of inconsistent ground truth.

How the pipeline works: three deterministic steps

Before getting into results, it's worth describing exactly what the system does. DataUnchain processor v2.0 implements a three-step pipeline. Two steps involve the vision model. One step is purely deterministic Python code.

Step 1 — Classify

The first step takes the document image (the PDF converted to a page image at 200 DPI using pdf2image and poppler-utils) and sends it to Qwen2.5-VL 7B running via Ollama with a classification prompt. The model must output one of the supported document type labels. No hints are given about what type the document might be — the model sees only the image and decides autonomously.

This matters in practice because real document workflows are mixed. A daily incoming-mail folder might contain invoices, delivery notes, payslips, credit notes, and contracts all in the same batch. The system must sort them correctly before it can extract anything. A classification error at this stage propagates to the wrong extraction prompt, producing garbage output.

Step 2 — Extract

Once the document type is identified, a type-specific extraction prompt is selected. Each of the seven supported document types has its own optimized prompt that describes exactly which fields to extract, in what JSON structure, with what handling rules for edge cases. For example, the invoice prompt specifies that if a tax line is present for a different VAT rate, it should be listed separately. The payslip prompt specifies that if multiple CCNL entries are present, the primary one should be selected. The bank statement prompt specifies that the transaction list should include date, description, debit amount, credit amount, and running balance for each entry.

The model returns a JSON object. This JSON is immediately validated against the schema for that document type. Missing required fields or type mismatches result in the document being flagged for human review before any further processing.

Step 3 — Audit

The third step involves no AI. A pure Python module runs formal validation and mathematical verification on the extracted JSON.

Formal validation includes: VAT number (P.IVA) checksum verification using the official 11-digit algorithm; Codice Fiscale format check (16 characters), pattern validation, check digit verification, and homocody-aware handling; date validation (ISO 8601 format, range 1900-2100, valid day for the given month/year); and numeric field validation (positive values, 2 decimal place precision).

Mathematical verification runs the appropriate check for the document type: taxable + vat = total ±€0.10 for invoices; gross - deductions = net ±€0.10 for payslips; opening + credits - debits = closing ±€0.10 for bank statements. The tolerance exists to accommodate rounding differences in the source documents.

The audit outputs a confidence score (HIGH, MEDIUM, or LOW) based on the overall internal consistency of the extracted data, and an audit_status: VALIDATED means the document can flow through automated processing; PENDING_REVIEW means minor issues were found that warrant a human glance; NEEDS_REVIEW means significant problems were detected and human intervention is required before the data is used.

Speed and throughput

At 32 seconds per document average, DataUnchain processes approximately 112 documents per hour on this hardware. The minimum observed was 25.8 seconds (simple, clean delivery notes), the maximum 53 seconds (complex bank statements with many transaction rows). The 90th percentile is 41 seconds, meaning 90% of documents finish within 41 seconds — useful for planning batch processing windows.

32 seconds may sound slow compared to traditional OCR, but the comparison is not valid. An OCR system extracts raw characters and layout coordinates. It has no concept of what fields mean, cannot validate a VAT number checksum, and certainly cannot verify that taxable + VAT = total. What DataUnchain does in 32 seconds — identify the document type, extract all semantic fields, validate fiscal identifiers, verify arithmetic — would take a human operator 2 to 5 minutes per document even working efficiently. At a cost of $0.002 per document at cloud rates, or essentially free on owned hardware, the economics are immediate.

What these numbers actually mean

VAT numbers and fiscal codes: 100% on 206 documents. This is the most operationally significant result in the entire benchmark. The Partita IVA and Codice Fiscale are the primary identity fields in Italian business documents. They determine who issued an invoice, who it was issued to, who signed a contract, whose payslip this is. If these fields are extracted incorrectly, the downstream consequence is not a minor inconvenience — it's wrong accounting entries, wrong tax reporting, wrong payroll records. Getting them right 100% of the time, across all document types, across both pristine PDFs and low-quality scanned documents, is the baseline requirement for any production deployment.

Financial amounts: 100% on every type that has them. Taxable amount, VAT, invoice total, net pay, bank statement closing balance — every single numeric financial field extracted is correct within the tolerance. This includes documents with amounts near psychological threshold values like €999.99 or €9,999.00, documents where the thousand separator could cause OCR confusion, and documents where JPEG compression has degraded the legibility of digits. The model handles all of these correctly.

Math check: 100% on 120 verifications. Not only are the individual financial fields correct, but they are internally consistent. Every invoice where we verify that taxable + VAT = total passes. Every payslip where we verify gross - deductions = net passes. This means the model reads the numbers coherently across the document — it doesn't read the taxable amount from one line and the total from another line of a different document, for example. The consistency check catches a whole class of subtle extraction errors that field-by-field accuracy metrics alone would miss.

The single critical failure: gross pay on payslips, 54.3%. Of 35 payslips in the corpus, 19 have the gross pay field correctly extracted and 16 do not. Examining the failures reveals a consistent pattern: the problem is not reading the number. When the model finds the correct field, the value is always right. The problem is identifying which of several differently-labeled fields represents the gross pay concept.

Italian payslips vary significantly depending on the Contratto Collettivo Nazionale di Lavoro (CCNL) — the collective bargaining agreement — and the payroll software used. The field labeled "retribuzione lorda" in a metalworking CCNL payslip is labeled "imponibile lordo" in a retail payslip, "totale competenze" in a construction payslip, and "imponibile contributivo" in a healthcare payslip. The net pay field, by contrast, is almost always labeled "netto in busta" or "netto a pagare" — highly consistent across CCNL types — and extracts at 100%. The fix is straightforward: provide the extraction prompt with an explicit enumeration of the known label variants. This is planned for processor v2.1.

The scan-equals-clean result matters because the most common objection to AI document systems is: “It works on clean PDFs, but all our documents are scanned, stamped, slightly rotated, faxed, re-scanned.” That objection does not apply here. Qwen2.5-VL was trained on enormous quantities of real-world document images, which inherently include degraded, scanned, and low-quality documents. The result is built-in robustness to the exact conditions that break traditional OCR-based systems.

We are not claiming the system is immune to all possible degradation. A document scanned at 72 DPI with severe motion blur, or a fax from 1995 transmitted over a poor line, might produce different results. But the conditions we tested — the actual conditions in a modern Italian office with standard network scanners — produce performance identical to native digital documents.

Confidence distribution: the system knows when it's unsure

A production document automation system must do more than extract data. It must also communicate its own uncertainty reliably. A system that extracts data incorrectly without signaling the problem is dangerous — errors flow silently downstream. A system that correctly identifies its own uncertainty allows humans to review only the uncertain cases, capturing the vast majority of documents in automated flow while maintaining a human-in-the-loop safety net for edge cases.

What happened inside the GPU: resource consumption data

We monitored hardware resource consumption every 60 seconds using nvidia-smi throughout the benchmark. The data provides a precise picture of the pipeline's hardware profile, which is critical for capacity planning.

The most important finding from the resource data: the pipeline is 100% GPU-bound. CPU utilization averaged 4% throughout the entire benchmark. The processor was doing almost nothing — converting PDFs to images, calling Ollama's HTTP API, writing JSON files. All of the computationally intensive work happened on the GPU. This has a direct implication for hardware planning: adding faster CPUs, more CPU cores, or more CPU RAM does not improve throughput. Only the GPU matters.

The VRAM consumption of 13.3 GB out of 16 GB available is worth noting. The margin of 2.6 GB is sufficient for the image buffers and model context during normal inference, but it is genuinely tight. This is why the bank statement GGML crash described in the next section occurs specifically on this hardware — the combination of a large dense image and a long extraction prompt pushes the tensor allocations beyond what the remaining 2.6 GB can accommodate.

The power draw of ~68 W during inference, compared to 6-7 W at idle, represents a significant but reasonable energy cost for continuous operation. At typical European electricity prices, running this GPU 8 hours a day for 22 working days a month costs approximately €2-4 in electricity. This needs to be accounted for in on-premise TCO calculations, but it is negligible compared to the labor cost of manual data entry at equivalent throughput.

$0.002 per document: what this means for real businesses

At $0.24/hr with 32 seconds processing time per document on cloud compute, the cost math is straightforward: $0.24 / (3600 / 32) = approximately $0.002 per document. Let's translate that into real business scenarios.

What “completely offline” actually means

The offline operation mode of DataUnchain is not a secondary feature — it is the primary architectural choice that differentiates it from every major commercial document extraction service. When we say the system runs completely offline, the implications are concrete and verifiable:

No data leaves your infrastructure. PDFs are converted to images locally. Ollama runs locally. The extracted JSON is written locally. Not a single byte of your documents — not metadata, not page thumbnails, not text fragments — is transmitted to any external service. Not to Anthropic, not to OpenAI, not to Microsoft Azure, not to Google Cloud, not to any other AI provider.

GDPR compliance is fundamentally simplified. The most complex GDPR obligations for organizations using AI services involve international data transfers, data processing agreements with AI vendors, ensuring that the AI provider handles your data according to GDPR requirements, and managing breach notification obligations to external processors. When your documents never leave your infrastructure, none of these obligations apply. Your data protection officer will appreciate this enormously.

Air-gap operation is possible. Once Qwen2.5-VL 7B is downloaded (approximately 5 GB, a one-time operation), the entire system runs without any internet connectivity. This enables deployment in environments that are genuinely isolated from the internet: manufacturing plant operational technology networks, secure government archives, legal document management systems with strict information barrier requirements, healthcare data systems subject to specific regulatory constraints.

No vendor lock-in, no subscription cliff. The underlying model (Qwen2.5-VL) is open source under Apache 2.0 license. Ollama is open source. DataUnchain is a commercial product built on top of these open foundations — you pay for the product, not for access to AI infrastructure controlled by someone else. If you want to switch to a different compatible VLM, that's a single configuration change. No SaaS vendor can raise prices on you, discontinue a tier you depend on, or sunset a feature that your workflow relies on.

Why Qwen2.5-VL 7B

The choice of Qwen2.5-VL as the backbone for DataUnchain's processor is based on systematic evaluation of the available open-source vision-language model landscape for document processing specifically.

LLaVA and its variants were the first widely-adopted open-source VLMs, but they show significant performance degradation on documents with dense structured text, tables, and multi-column layouts. Their training data skewed heavily toward natural images, and document understanding was not a primary design objective.

InternVL2 shows strong document understanding performance but has a less mature deployment ecosystem. Integrating it into a production pipeline requires more custom work compared to the Ollama-based deployment that Qwen supports natively.

Qwen2.5-VL from Alibaba DAMO Academy was specifically designed with document understanding as a first-class capability. Its training data includes large quantities of structured documents from multiple languages, including Italian business documents. The model demonstrates particularly strong performance on tasks requiring spatial understanding of form layouts, table extraction, and recognition of language-specific fiscal identifiers. The 7B size hits the sweet spot between capability and hardware accessibility: it runs on a 16 GB GPU — within reach of many organizations — while delivering accuracy that this benchmark demonstrates is production-grade on most document types.

It is also worth being explicit about what Qwen2.5-VL is not: it is not an OCR system. OCR converts pixel patterns to characters without any understanding of what those characters mean. Qwen2.5-VL is a multimodal language model that genuinely comprehends documents: it understands that a number after “P.IVA:” is a VAT identifier, that a row in a table with a date and a euro amount followed by “D” is a debit transaction, that text in a box labeled “NETTO IN BUSTA” at the bottom of a page is the net pay figure. This semantic understanding is what enables 100% accuracy on fiscal identifiers without requiring a separate regex post-processing layer.

Hardware deployment guide

Based on the resource consumption data from this benchmark, here are concrete hardware recommendations for production deployment:

Results by document type

Benchmark methodology

Every result published here was produced by a fully automated pipeline with no manual intervention in the evaluation step. The evaluation process consists of four stages: document generation with fixed random seed; processing through DataUnchain's processor v2.0; automated field-by-field comparison against ground truth; and aggregation into the final report.

Numeric fields are evaluated with a tolerance of ±€0.50 to account for rounding conventions. Date fields require exact match in ISO 8601 format (YYYY-MM-DD). String fields (VAT numbers, fiscal codes, document references) require exact match. Classification is evaluated as correct or incorrect with no partial credit.

The benchmark methodology is fully documented. If you want to validate these results on your own document corpus as part of a proof-of-concept engagement, contact us — we run structured pilots with prospective clients on their own documents under NDA.

What comes next

The two identified limits have concrete, planned fixes. The payslip gross pay prompt enrichment is the simplest and highest-priority change — a straightforward update to the extraction prompt with all known CCNL label variants for the gross pay field. We expect this to bring accuracy from 54.3% to above 90% based on the pattern of failures.

The bank statement GGML crash fix involves implementing adaptive DPI reduction: the processor will detect when a document page contains more than a threshold number of text elements (via a quick image density metric), and automatically reduce the conversion DPI from 200 to 150 for those pages. This reduces the image size enough to keep tensor allocations within the 16 GB VRAM budget while maintaining sufficient resolution for reliable text recognition.

The benchmark v3 will extend the corpus to ten document types, adding receipts and commercial documents, packing lists, quotations, and healthcare-specific formats. The target is 300 documents across the full range. We will also run comparative benchmarks against Amazon Textract, Azure Document Intelligence, and Google Document AI on the same corpus to give the market objective comparison data.

A test of Qwen2.5-VL 32B on a 48 GB GPU is planned to quantify the accuracy delta between model sizes. The hypothesis is that the 32B model resolves both the bank statement GGML issue and the payslip gross pay label variance through its larger and more capable vision encoder.

The bottom line

95.5% accuracy. $0.002 per document. 32 seconds. Zero cloud. Zero data leaving your infrastructure.

On the fields that matter most for Italian business automation — VAT numbers, fiscal codes, dates, financial amounts, arithmetic consistency — the system achieves 100% on every one. Scanned documents perform identically to native digital PDFs. The system communicates its own uncertainty rather than silently inserting bad data into downstream systems. Two identified limits are fully understood and have clear, planned fixes.

95.5% accuracy on a corpus of 219 real Italian business documents, with 100% on the fields that matter most for automation: VAT numbers, fiscal codes, dates, amounts, and arithmetic consistency. This is the bar we hold ourselves to before calling a system production-ready.