AI Document Ingestion
for Enterprise Systems

Every business day, millions of documents land in corporate inboxes: purchase orders from suppliers, invoices to be approved, delivery notes to be matched, contracts to be filed, customs declarations to be archived. Each one demands human attention. Someone must open it, read it, type data into a system, verify the numbers, and move on to the next. It sounds routine. Multiplied across an organisation, it is one of the most expensive and error-prone activities in modern business operations.

Research consistently shows that the average enterprise knowledge worker spends 4 to 6 hours per week on manual data entry and document-related administrative tasks. In a company of 50 people, that is up to 300 person-hours every week spent on work that, in principle, a well-designed system could handle automatically. Annually, that scales to over 15,000 hours — the equivalent of 7 to 8 full-time employees working on nothing but copy-pasting data.

The direct labour cost is only part of the picture. The deeper issue is error propagation. When a human types a supplier's VAT number incorrectly — transposing two digits under pressure to clear a backlog — the downstream consequences can be severe. The invoice fails fiscal validation. The payment is blocked. The supplier relationship is strained. The accounting team spends hours reconciling what went wrong. A single mistyped VAT number can hold up a payment worth tens of thousands of euros and trigger a compliance audit.

The Document Flood

Modern businesses receive documents in every conceivable format and through every conceivable channel. Invoices arrive as PDF email attachments, scanned paper pushed through a multifunction printer, EDI messages from large-enterprise trading partners, and increasingly as structured XML files via certified digital channels like Italy's Sistema di Interscambio (SDI). Contracts arrive as Word documents, then as signed PDFs, then sometimes re-scanned after wet signatures. Delivery notes (DDTs in Italy) come bundled with shipments or emailed separately. Purchase orders originate in the company's own ERP but need to be matched against supplier confirmations.

Each document type has a different layout. A single document type — say, a supplier invoice — can appear in thousands of different visual formats, one per supplier, each with vendor name, date, amounts, and line items placed wherever the supplier's ERP or design team decided to put them. Traditional software approaches this problem with templates: one template per supplier layout. When the supplier updates their invoice design, the template breaks. The document fails. Someone has to fix the template.

Compliance Risk and Late Payments

Beyond efficiency, there is a compliance dimension. GDPR mandates that personal data — present in HR documents, medical reports, contracts — be handled with documented controls and limited access. When documents sit in a shared network folder waiting for manual processing, they are effectively uncontrolled. Who accessed the invoice containing personal bank details? Nobody knows. The audit trail does not exist. Regulatory risk accumulates silently.

Late payments have direct financial consequences. The EU Late Payment Directive and equivalent national legislation entitle creditors to interest on overdue invoices. Companies that fail to process invoices promptly — because humans can only work so fast — routinely incur penalty interest costs and damage supplier relationships that took years to build.

AI document ingestion is the automated process by which a software system receives unstructured or semi-structured business documents, applies artificial intelligence to understand their visual and semantic content, extracts structured data fields, validates the results against business rules and mathematical constraints, and routes the validated data to downstream systems such as ERP platforms, CRM databases, or workflow automation tools — without requiring human intervention for routine cases.

This definition distinguishes AI document ingestion from three related but distinct concepts: optical character recognition (OCR), robotic process automation (RPA), and traditional document management systems (DMS).

How it Differs from OCR

OCR (optical character recognition) is a prerequisite technology, not a solution. OCR converts the pixels in a scanned image into machine-readable text. It does not understand the text. It does not know that "€ 1.234,56" is a total amount, or that "IT12345678901" is a VAT number, or that the text in the upper right corner is the invoice date while the text in the lower right is the payment due date. OCR hands you a stream of characters. A human — or a rigid rules engine — still has to interpret what those characters mean.

AI document ingestion goes further. The AI model reasons about the document as a visual artefact: it understands layout, context, relative position, and semantic meaning simultaneously. It can look at an invoice it has never seen before and correctly identify the supplier name, total amount, VAT amount, and line items — not because it has a template for that specific supplier, but because it understands what invoices look like and how their components relate to each other.

How it Differs from RPA

Robotic process automation (RPA) automates human interactions with software interfaces: it clicks buttons, fills forms, copies values from one application to another. RPA is powerful for automating structured, deterministic processes — but document processing is inherently unstructured. When you point RPA at a document, you still need something to extract the data first. And when you use template-based extraction, the RPA workflow breaks the moment the document layout changes.

AI document ingestion replaces the fragile template-matching step with a model that generalises across layouts. RPA can then be used for the final step — pushing validated data into a target system — but the understanding layer is AI, not rigid rules.

The Five Core Steps

┌─────────────────────────────────────────────────────────────┐
│                     DOCUMENT SOURCES                         │
│  Email/IMAP  │  REST API  │  Telegram Bot  │  Folder Watch  │
│  SDI / PEC Monitor                                          │
└──────────────────────────┬──────────────────────────────────┘
                           ↓
┌──────────────────────────────────────────────────────────────┐
│                    INGESTION LAYER                            │
│  File type detection │ PDF rendering │ Multi-page split       │
│  Deduplication hash  │ Quarantine queue │ Job scheduler       │
└──────────────────────────┬───────────────────────────────────┘
                           ↓
┌──────────────────────────────────────────────────────────────┐
│                AI UNDERSTANDING LAYER                         │
│  Vision-Language Model: Qwen 2.5-VL (local Ollama)           │
│  Document classification │ Entity extraction │ JSON output    │
│  Confidence scoring per field                                 │
└──────────────────────────┬───────────────────────────────────┘
                           ↓
┌──────────────────────────────────────────────────────────────┐
│                  VALIDATION LAYER                             │
│  Math: subtotal + VAT = total (±€0.02 tolerance)             │
│  Format: IT VAT numbers, fiscal codes, IBAN, dates           │
│  Confidence threshold routing → VALIDATED / NEEDS_REVIEW     │
│  Audit log entry creation                                     │
└──────────────────────────┬───────────────────────────────────┘
                           ↓
┌──────────────────────────────────────────────────────────────┐
│                 INTEGRATION LAYER (18 adapters)               │
│  CRM:     Salesforce │ HubSpot │ Airtable │ Notion           │
│  ERP:     SAP B1 │ Odoo │ Zucchetti │ TeamSystem │ Mexal     │
│  Files:   CSV │ Excel │ FatturaPA XML                        │
│  Notify:  Email SMTP │ Slack │ Microsoft 365                 │
│  Custom:  Webhook │ RPA Playwright                           │
└──────────────────────────────────────────────────────────────┘
        

Layer 1: Document Sources and Ingestion

The ingestion layer is responsible for accepting documents from all configured input channels and normalising them into a common format for downstream processing. Email monitoring uses IMAP IDLE to receive attachments the moment they land in the inbox — no polling interval, no delay. The folder watchdog uses operating system file system events rather than polling, achieving sub-second detection latency even on network-mounted volumes.

Once a file is received, it is assigned a SHA-256 hash for deduplication: if the same document arrives twice (common with email forwards), only one copy is processed. PDF rendering converts each page to a high-resolution PNG (typically 300 DPI) to preserve fine detail such as small-print legal clauses, handwritten annotations, and low-contrast watermarks. Multi-page PDFs are split intelligently: a five-page invoice is processed as five separate images, then the results are merged back into a single structured document.

Files that cannot be parsed — password-protected PDFs, corrupted archives, unsupported binary formats — are moved to a quarantine queue and a notification is sent to the operations team. Processing never silently fails.

Layer 2: AI Understanding with Qwen 2.5-VL

The AI understanding layer is the core intellectual component of the system. DataUnchain uses Qwen 2.5-VL, a state-of-the-art vision-language model that processes document images directly, without first running OCR. The model runs locally via Ollama — it never communicates with external servers. Inference happens on the same machine that received the document.

The model receives a carefully engineered prompt alongside the document image. The prompt instructs the model to identify the document type (invoice, contract, DDT, medical report, pay slip, etc.), extract all relevant fields with their values, and express confidence scores for each extraction. The output is a structured JSON object that downstream layers can process programmatically.

Qwen 2.5-VL's vision capability means it can handle documents that would defeat a pure-text approach: tables whose columns are defined by visual alignment rather than delimiters, handwritten annotations in the margins, dual-language headers, stamps and watermarks overlapping text, and mixed-orientation pages where some sections run vertically.

Layer 3: Validation

AI extraction is probabilistic. The validation layer applies deterministic rules to catch the errors that the AI might make. Mathematical validation checks that subtotal + VAT = total within a tolerance of €0.02 (to account for rounding artefacts in supplier accounting software). If the numbers do not reconcile, the document is flagged NEEDS_REVIEW rather than auto-dispatched.

Format validation covers Italian-specific requirements extensively: VAT number (Partita IVA) format and checksum, fiscal code (Codice Fiscale) with full omocodia support (the alternative encoding used when a standard fiscal code is already taken), IBAN check-digit validation, and date range plausibility checks (an invoice dated three years in the future is a data error, not a valid document).

The confidence threshold system assigns each document to one of three audit statuses: VALIDATED (all checks pass, high confidence — auto-dispatched), PENDING_REVIEW (medium confidence — queued for operator confirmation), or NEEDS_REVIEW (failed validation — flagged for correction). This routing logic ensures that automation never silently produces wrong data: it either gets it right and routes automatically, or escalates to a human.

Layer 4: Integration and Output Adapters

The integration layer dispatches validated data to one or more output adapters. Each adapter is an independent module with its own connection configuration, retry logic, and error handling. The webhook adapter can call any REST endpoint, making it compatible with virtually any modern SaaS platform. The FatturaPA XML adapter generates legally compliant Italian electronic invoices. The RPA Playwright adapter drives a browser to interact with legacy web applications that have no API.

All adapter outputs are logged with timestamps, HTTP status codes, and response payloads. If an adapter fails — network timeout, authentication error, target system downtime — the document enters a dead-letter queue. The system retries with exponential backoff. After a configurable number of retries, the document is escalated to the operations team via notification. No data is silently lost.

Why OCR Alone Fails on Real-World Documents

The fundamental limitation of OCR is that it converts pixels to characters without understanding context. A typical supplier invoice contains dozens of numbers: the invoice number, the order reference, the supplier's VAT number, the buyer's VAT number, the net amount, the VAT amount, the total amount, the payment due amount, line item quantities, line item unit prices, and various reference codes. OCR produces all of these as text strings. Without a way to understand which string is which, you have meaningless data.

The traditional solution is template matching: define the coordinates on the page where specific fields appear, then read the value at those coordinates. This works for a single supplier whose invoice layout never changes. Real enterprise procurement teams deal with hundreds of suppliers. Each supplier has their own layout. A company that buys from 200 suppliers needs 200 templates. When any of those suppliers changes their invoice design — and they do, regularly — the template must be updated manually. The maintenance burden grows faster than the business.

Beyond templates, real-world documents are additionally challenging because of scan quality: skewed pages, coffee stains, faded ink, mixed black-and-white and colour content, rubber stamps overlapping printed text, and handwritten corrections to typed fields. Tesseract and similar engines handle ideal images adequately; they degrade badly on anything that deviates from clean, horizontal, high-contrast text.

Why RPA is Fragile for Document Processing

RPA tools excel at automating deterministic, UI-driven workflows: "click here, type this, save that." The fragility emerges when the input is unstructured. Document-centric RPA workflows inevitably include a data extraction step, and that step is almost always template-based OCR or pattern matching. This makes the workflow's reliability entirely dependent on the reliability of the extraction step.

When a document arrives with an unexpected layout — a supplier who changed their invoicing software, a scanned document that is slightly rotated, a PDF with embedded images rather than native text — the extraction step fails. The RPA bot either throws an error (good outcome — at least it is visible) or silently extracts the wrong values (catastrophic — the wrong data enters your ERP without any alert).

Why Cloud AI Creates Privacy and Compliance Problems

Services like GPT-4V, Google Document AI, and AWS Textract solve the understanding problem well. They are genuinely capable of reading and interpreting complex documents. The problem is what happens to the document after you send it. These services process your data on their infrastructure. For routine business documents, this may be acceptable. For documents containing personal data — pay slips, HR contracts, medical records, legal agreements — sending the document to a third-party cloud service raises immediate GDPR compliance questions.

The controller-processor relationship must be documented in a Data Processing Agreement (DPA). Sub-processors must be listed and consented to. Data transfers outside the EU require additional safeguards under Chapter V of the GDPR. Any data breach at the cloud provider must be reported. For companies in regulated sectors — healthcare, finance, legal, government — these requirements often make cloud AI processing of sensitive documents legally untenable or operationally impractical. Local AI processing eliminates this category of risk entirely.

Why GDPR-Conscious Companies Can't Send Documents to the Cloud

The General Data Protection Regulation establishes specific obligations whenever personal data is transferred to a processor (Article 28) or to a third country (Chapter V). Most AI document processing services are operated by US-based companies. Sending a European citizen's pay slip, medical report, or employment contract to a US-based AI API constitutes a personal data transfer that requires legal safeguards — Standard Contractual Clauses (SCCs), Binding Corporate Rules (BCRs), or an adequacy decision.

Even with SCCs in place, many data protection officers (DPOs) are uncomfortable with the practical reality: the document is processed on servers outside the organisation's control, by a service whose sub-processors may change, in a jurisdiction where government access to data may occur without notification. For organisations in healthcare, legal, financial services, or government, this is often categorically unacceptable.

Architecture of a Zero-Egress System

DataUnchain is designed as a zero-egress system by architecture, not by policy. No document pixels, no extracted text, no intermediate processing artifacts ever leave the machine (or the local network) on which the system runs. The AI model (Qwen 2.5-VL via Ollama) runs as a local process. API calls are made only to the local Ollama server's loopback address. The integration adapters send structured data (field values, not document images) to the configured output systems — and even these can be restricted to internal systems only.

The system can operate in a fully air-gapped configuration: no internet connectivity at all. In this mode, all software updates are applied via physical media, and all output goes to on-premise systems (ERP, local database, network share). This configuration is suitable for classified environments, high-security industrial settings, and healthcare networks that operate behind strict network perimeters.

What "Local LLM" Means in Practice

Running a large language model locally means that the model weights — the trained neural network parameters — are stored on the local machine's GPU or CPU memory. When a document is processed, the model is invoked directly via the Ollama API on localhost, produces its response, and that is the end of the inference step. No network call is made. No telemetry is sent. The model has no mechanism to exfiltrate data because it is not connected to anything outside the host machine.

Qwen 2.5-VL is an open-weights model published by Alibaba Cloud and downloadable from Hugging Face. The weights can be verified by hash. The inference stack (Ollama) is open-source and auditable. Organisations with security requirements can conduct their own code review of the entire stack that touches their documents.

The Most Common Failure Modes

Building a document ingestion system that works in a laboratory against clean, well-formatted PDFs is straightforward. Making it reliable against the full chaos of real enterprise documents is a different problem. The failure modes we encounter most frequently:

Why Confidence Scoring Matters

AI extraction is not binary — it is not either correct or incorrect. The model produces a result along with an estimate of how confident it is in that result. A confidence score of 0.98 for a VAT number extraction means the model is highly certain. A score of 0.61 means the model extracted something, but it is not sure. These are very different situations, and treating them identically — auto-dispatching both — would be a design error.

The confidence threshold system routes documents based on their aggregate confidence profile. A document where all fields score above 0.90 is auto-dispatched. A document with one field scoring below 0.75 is queued for human review of that specific field. This means the human reviewer sees only the uncertain extractions, pre-highlighted, rather than having to re-read the entire document. Review time is measured in seconds, not minutes.

Over time, patterns in review decisions can be used to improve the system. If reviewers consistently correct "Importo Netto" to be treated as the subtotal rather than the total, that correction can be fed back into the prompt or post-processing rules. The system learns from exceptions without requiring full model retraining.

The Importance of Human-in-the-Loop

A common mistake in automation projects is designing for the average case and leaving edge cases unhandled. In document processing, edge cases are not rare — they are a predictable percentage of every document stream. A system without a well-designed human review workflow will either fail silently (processing errors go unnoticed until month-end reconciliation) or require someone to manually check every document regardless of quality (defeating the purpose of automation).

Human-in-the-loop design means: documents are routed to human review when they need it, the review interface shows exactly what needs attention, the reviewer's corrections are recorded, and the corrected output proceeds automatically. The human is a quality gate, not a processing step. This design allows automation to handle 85-95% of documents fully automatically while maintaining 100% accuracy across the board.