Build note: ingesting 139,000 documents nobody had organised

Most AI projects in established businesses look like a data problem wearing a modelling costume. The interesting work is rarely the model. It is getting a decade of accumulated mess into a shape the model can be useful against.

This is a note on one of those jobs: an Australian professional-services firm whose entire working history lived in per-client folders on a file server. 3,091 client matters. 114,107 emails. 139,388 documents, PDFs and images. All of it real, none of it organised for retrieval.

The shape of the mess

The file store was organised the way every file store is organised: by client, then by whatever the person handling that client felt like doing. That structure answers exactly one question — what do we have for this client? — and no others. It cannot answer have we handled this situation before?, which is the question that actually has money attached to it.

Three specific problems mattered more than the volume:

Emails carried the reasoning. The documents were artefacts; the thinking lived in the correspondence around them. Any system that indexed only the attachments would miss why anything was done.

A large share of the documents were images. Scans, photographs of paperwork, screenshots. To a text pipeline these are blank pages. Ignoring them would have silently dropped a meaningful fraction of the firm's history — the worst failure mode, because the system would still look like it worked.

Nothing was uniform. Same document type, twenty years, a dozen staff, several naming conventions, and every generation of file format that implies.

The pipeline

Extraction ran per matter rather than per file type, which mattered for keeping a matter's material together and for making the run resumable — 3,091 matters is a lot of opportunities to crash at document 90,000.

Each matter came out as three structured notes: a case story summarising it, a correspondence timeline built from the email bodies, and a document register holding the extracted text of everything filed. That three-part shape turned out to be the useful unit. Retrieval nearly always wants one of those three things, not the raw folder.

OCR ran locally over everything image-shaped, writing results to a sidecar file next to each matter's register and caching aggressively. That cache is now around half a gigabyte, which sounds wasteful until the first time you need to re-run the pipeline and don't have to re-OCR 100,000 images.

What actually cost time

OCR, by a wide margin. Not the accuracy — the throughput, the supervision, and the long tail of files that hang, crash, or produce garbage. It needed a supervisor script and several restarts. Budget for this honestly: in a document-heavy business, OCR is not a checkbox on the plan, it is a phase of the plan.

Deciding what not to extract. The temptation is to capture everything. But every additional extracted field is something that can rot, and much of what sits in a file store is noise — duplicate drafts, mail signatures, scanned envelopes. Indexing noise doesn't just waste space; it degrades every search that follows.

Resumability. Any pipeline at this scale will be interrupted. Designing for that from the start is cheaper than discovering it at 70% through a nine-hour run.

What I'd tell a business considering this

Three things I now say in the first conversation:

Your data is worse than you think, and that's normal. Every business I've worked with believes its filing is roughly fine. It never is. This isn't a criticism of the business — it's just what a decade of real work under time pressure looks like. Plan for it instead of being surprised by it.

Look at the real data in week one. A pilot on a clean sample tells you almost nothing, because the sample won't contain the photographed document, the mis-scoped folder, or the matter someone handled differently in 2019. Those are the whole job.

The index is the asset, not the AI. Once the history is extracted, structured and searchable, you can point whatever model you like at it — and swap that model next year. Businesses that spend their budget the other way round, on the model layer over unstructured mess, end up with a demo instead of a system.

The search and retrieval layer built on top of this is its own note. The architectural decision that made the whole thing safe to use in a regulated practice is the two-layer split.

---

Written with the firm's identifying details removed.