CASE FILE

CH.08APPLIED AI

Multimodal Document Intelligence

OCRVLMLORAHUMAN-IN-THE-LOOP·  LID VIZION
A mechanical eye on an articulated arm inspects floating documents while glowing glyphs lift off the paper as light.

THE PROBLEM

At Lid Vizion, ops reviewers were hand-keying structured fields from documents that arrive photographed, skewed, stamped, and half-legible — and volume had outgrown the team. Off-the-shelf OCR read the characters and missed the meaning, and my first automated version did the worst thing an extraction system can do: it passed confident wrong fields downstream silently. I rebuilt it so the pipeline scores its own confidence and routes the hard cases to the humans it was displacing badly.

CONSTRAINTS

  • Documents arrive as degraded photos, not clean scans
  • Field layouts vary across every source
  • Wrong extractions are worse than no extractions — learned that live
  • Human review time is the scarcest resource

DECISIONS & TRADEOFFS

FINE-TUNED VLM over pure OCR plus rules — because layout and semantics carry the answer on degraded documents, and LoRA gets a task-specialized model without retraining a giant

CONFIDENCE-RANKED REVIEW QUEUE over full automation — because routing low-confidence extractions to humans keeps accuracy where the business needs it while review effort shrinks to the hard cases

CV PREPROCESSING over raw-image inference — because deskewing and quality checks before the model turn garbage-in into a detectable, routable event instead of a silent wrong answer

ARCHITECTURE

01DOCUMENT INTAKEphotos, scans, mixedquality02CV PREPROCESSINGdeskew, denoise, qualitygate03OCR + VLMlora-tuned extraction04CONFIDENCE SCORINGper-field certainty05REVIEW QUEUEhumans on low-confidenceonly06STRUCTURED OUTPUTvalidated fieldsdownstream

OUTCOME

Fields come out structured with a confidence score attached, so downstream systems know what to trust instead of trusting everything. Reviewers stopped keying and started judging only the cases the model is unsure about, which is a different job and a shorter queue. Every human correction feeds back into the eval set, so the queue keeps shrinking for real reasons.

PROOF / LINKS

PROPRIETARY WORK — THE SHAPE OF IT IS PUBLIC HERE, THE SYSTEM ITSELF IS NOT.

AT 100× SCALE

At 100x this becomes a self-improving extraction service: corrections continuously retrain the adapter, confidence thresholds auto-tune per document type, and human review concentrates on genuinely novel layouts.