Reading the loan file: 109 questions, 600-page scans, US$3.56 a document
Reading the loan file
A commercial property loan is reviewed against its paperwork. Valuations, facility agreements, tenancy schedules, rent rolls and covenant workbooks arrive as a pack of 20 or 30 files per loan. An analyst reads the pack and answers the same 60 or 70 questions every time. Who are the tenants? When do their leases end? What is the weighted average unexpired lease term? Which valuation is the current one? Today that review happens once a year. The bank wanted it every quarter, across about 200 loans.
We designed and built the pipeline that reads the pack. It turns a folder of mixed documents into one answer per question, per property and per tenant, each with a confidence and the passage it came from. It runs on Google Cloud with Gemini models. On a production-scale run it cost about US$3.56 per document, all in.
The challenge
The documents do not cooperate. The largest single file in one run was a 618-page valuation, much of it scanned. The smallest were one-page letters. Rent rolls arrive as spreadsheets with a banner title where the header row should be, with properties down the side and months across the top, or with the whole table turned on its side. The same cash-flow workbook arrives again every month with small changes. One pack held a workbook with 36 sheets.
The names do not cooperate either. One property can appear as a unit number, a street address and a title number in three different documents. A tenant is a trading name in the lease and a company name in the rent roll. Until the system knows that these are the same entity, it cannot answer a question about it.
And the questions are not all look-ups. We sorted the 109 questions in the lending question set by what it takes to answer them:
- 44 are look-ups: find the value in a document.
- 10 need the latest version: an older document states a value that a newer one replaces.
- 34 need identity resolution: the answer depends on knowing which mentions refer to the same property or tenant.
- 21 are calculations over many rows, such as a weighted average across every lease.
About 37% of the questions cannot be answered reliably by a naive agent or by retrieval over chunks. Neither approach can produce the audit trail a credit review needs for any of them.
Why it matters
A credit analyst signs their name under the answers. An answer without a source is a guess. An answer from the wrong valuation is worse than no answer, because it looks right. And a calculated figure, such as a weighted lease term, must be calculated. A model's estimate of an average is not an average.
Cost matters too, in a specific way. The review runs every quarter over thousands of documents. A design that costs a few dollars more per document is a line in the budget. A design whose cost nobody can predict does not get approved.
The approach
The whole pipeline makes two moves. First, find out who the entities are: the loan, its properties, its tenants. Give each one a single canonical name and record every alias. Second, give each question only the documents in which its entity appears, and no more.
Everything before the final answer exists to make that second move precise. This is retrieval by entity, not by similarity. We looked hard at embedding search and rejected it as the main route. It is cheaper, and it is worse. It finds passages that sound like the question, but the question needs passages about the right tenant.
The question set is the configuration
The questions arrive as a YAML file, and the pipeline derives the rest from it. A first pass reads the question set and builds the entity hierarchy: loan, then property, then tenant. That hierarchy sets what the classifiers look for, what the extractors fill in and how the final work is planned. A different question set produces a different hierarchy. We proved this on a completely different domain, a cookbook, where the same pipeline built country, region and recipe from a different question set.
Code first, a model for what code cannot settle
The pipeline has 18 phases, in three tiers. Every phase is tagged: plain code, a model call, or code first with a model as the fallback. That tagging is the design rule of the whole system. Deterministic code runs first wherever it can. The model handles only what code cannot settle.
The examples run through every tier. Aliases merge by a union-find graph before a model is asked to confirm anything. The first deterministic registry merge ran in 0.31 seconds with no model calls at all, about 97 times inside its time budget. Tenants attach to properties by a postcode join before a model sees the edges the join could not close. And calculated answers come from a formula engine, not from a model.
Every page chooses its own reader
Everything becomes markdown early. Markdown keeps the heading structure, and the structure is what the later phases use to place each fact in context. File type is read from the first bytes of the file, not from its name.
A PDF is not one kind of document. It is a stack of pages, and each page can be different. Code classifies every page with no model call. A page with fewer than 50 characters of text is a scan. A page with text and an image that covers more than 80% of it is a hybrid, such as a photographed table with a caption. Anything else is native.
Native pages are read straight from the text layer, for free. Scanned and hybrid pages go to a vision model, and the details there are where the engineering lives:
- Pages render at 300 DPI, capped at 12 megapixels, so an oversized drawing cannot flood the model with tokens.
- Pages travel in batches of five, four batches at a time. Each worker renders its own pages. Peak memory depends on the batch size and the number of workers, not on the length of the document, so a 600-page scan needs no more memory than a 6-page one.
- The model returns markdown with its headings in place, so a scanned page enters the structure parser exactly like a native one.
- Code splits each batch reply on a page-break marker. If the split fails, the batch falls back to one call per page. A page that still fails is retried once at 150 DPI.
- Every page records its own tokens, cost, time and model.
Spreadsheets are read by code
A rent roll is a table, and a table has a structure that code can read. Sending it row by row to a model is slow, expensive and less reliable.
A workbook is converted to markdown in a separate process, with empty columns removed and a marker for each sheet, so a sheet boundary survives everything that follows. Images inside the workbook are extracted with their cell positions. One vision prompt does two jobs: it transcribes an image that is a scanned page or a receipt, and it describes an image that is a chart or a photograph. Each result goes back under its own sheet, in row order.
Code then scores every table for regularity: how many rows are sparse, and how consistent each column's data type is, against one calibrated threshold. If every table in a sheet is regular, one model call maps the columns to the fields in the schema, and code applies that mapping to every row. A 100-row rent roll costs one call, not a hundred. A column the model is unsure of is dropped, never guessed. Only an irregular sheet goes to a model, in bounded windows of rows. Known sheet layouts match saved profiles, cached by the shape of the sheet, and sheets turned on their side have their own reader.
Context without resending the document
Each section of a document gets a short label that says what it is about and which entity it concerns. A naive design sends the whole document with every section, so the model can see the context. For a 2,500-section document that is the whole document, 2,500 times.
We took the idea behind contextual retrieval and changed what counts as context. Each section's prompt carries a 50 to 100 token summary of the document and the chain of headings above the section. Nothing else. Our estimate at the time was US$50 to US$100 per large document the naive way, against US$5 to US$15 this way. Prompt caching attacks the same cost by storing the repeated context. This design attacks it by not sending that context in the first place.
Answers that can refuse
The final tier plans the work in dependency waves, because some answers need other answers first. Each task pairs one question with one entity, and it carries a filter of the documents in which that entity appears. The answer prompt lists the entity's aliases, so the model does not refuse because a lease uses a trading name. Every answer returns a value, a confidence, its reasoning and the source spans it used.
Calculated answers do not go to a model at all. They are declared in the question set in a small formula language:
operation: weighted_average
value: { op: date_diff_years, from: compute_as_at_date, to: { op: ref, attribute: lease_expiration_date } }
weight_by: rent_payable
min_coverage_ratio: 1.0
That is the weighted average unexpired lease term, in four lines. If any lease is missing a rent or an expiry date, the engine refuses to answer. It does not fall back to a model's guess.
Throughput across projects and regions
A quarterly run across 200 loans is hundreds of thousands of model calls. One cloud project in one region has a quota far below that. So every call goes through a router of our own that spreads work across several projects and regions.
The router admits calls by tokens per minute, not by request count, because one request can be a hundred tokens or a million. It estimates the size of each request from its byte length, so it never parses a multi-megabyte prompt on the path that accepts traffic. After the call it corrects the estimate against the true token count. Each call goes to the region with the most headroom. An early measurement found the pipeline using about 2% of the capacity it had. Closing that gap took changes to the message broker as well as the router.
Every stored result is keyed by a hash of the document's content, and every step checks for its own output first. Running a document again costs nothing for the work already done. Each file type has a deadline, and a reaper closes out any document that exceeds it, so one bad file cannot hold up a loan. Messages that keep failing go to a dead-letter queue. Derived data in Redis expires after 30 to 90 days, and the source documents remain the only record, which keeps questions of data ownership simple.
Making every call earn its place
The first priority was recall: find every fact, in every document, and lose nothing. May was the month we pushed that as far as it would go. The team scaled up development, ran the pipeline again and again against real loan packs, and let each stage do all the model work it could. The cost of that month made the price of every design choice visible, stage by stage.
Three weeks of cost engineering followed, in about 400 commits. Each change was measured against saved pipeline state from before and after, and each shipped with a regression test.
The biggest change was in attribution, the step that decides which entity a fact belongs to. It had been checking every candidate fact in a document against every entity. Grounding the candidates in the relationships the pipeline already knew cut them from about 1,251 per document to about 92. The facts that still needed a paid model call to settle fell from about 466 to about 21. Calls with nothing to decide, about 206 per document, went to zero.
The rest was the same discipline applied everywhere. A text value that does not appear in its source is rejected before it enters the registry. Spreadsheet mapping went from one call per chunk to one call per table. The most token-heavy stage was made safe to repeat, so a retry no longer ran it again. An empty model reply is no longer retried and billed. The largest-volume extraction stage moved to the faster, cheaper model. Vision tokens got their megapixel budget. And a run is only marked complete when every loan in it is.
The like-for-like comparison is the clearest result. In early May, a run over one pack of about 263 documents stalled at around 208 and had to run again in full. In June, the same pack ran to completion: 252 of 265 documents finished, across three loans, and the day's whole cloud bill was US$896.
The breakdown shows where the money goes. A PDF of one to three pages, the most common file, cost about US$0.63. A workbook cost about US$4.84. The 19 PDFs over 50 pages averaged about US$29 each and took 61% of the spend. Cost lives in a short tail, which makes it predictable and tells the bank exactly where to curate inputs. The faster model carried 84% of the bill and the larger model 16%. So the next saving is batch processing on the high-volume calls, at about half price, not another model change.
Entities the analyst would recognise. Before the registry work, one loan's pack produced 642 "properties" that were mostly noise. After it, the same pack produced 39 real development names. 1,304 tenant "names" that were row numbers became 1,013 real tenant identifiers. On one loan the pipeline found 53 of the 55 properties in the analyst's own records. A single fix to how questions reach their documents cut "no context found" answers from 47% to 8%.
Built to be checked. The repository holds 135 dated specifications and more than 4,600 tests. A separate evaluation project scores recall and precision per loan against ground truth for eight loans. It reads saved pipeline state and never runs the pipeline, so a score costs nothing. Integration tests replay recorded state from real runs.
Key decisions
A registry, not a search index. Most of the hard questions depend on identity: which valuation, which tenant, which property. Answering them needs a canonical list of entities, built once, not a search that runs again for every question.
Code first, and the model as the fallback. Code is repeatable, free and testable. The model is flexible. Every phase uses code for what code can decide and the model for the rest, and the phase is labelled so that everyone knows which is which.
Route pages and sheets, not files. A PDF is a stack of pages that need different readers. A workbook is a set of tables that code can read once it knows the columns. The unit of routing is the smallest unit that behaves differently.
Refuse rather than estimate. A calculated answer with missing inputs returns no answer. A column the model is unsure of is dropped. A text value not found in its source is rejected. In a credit review, a gap that is visible is safe, and a plausible number is not.
Measure cost per document, per stage. Monthly spend hides the cause. Cost per document by type, tokens per stage and calls per document show exactly which design choice costs what. The direction that followed is written down: correct answers first, then batch pricing and a cost ceiling per run. Optimising the cost of a wrong answer is the wrong optimisation.
Lessons
The document is not the unit of work. Pages, sheets, tables and sections behave differently. Route each one to the cheapest reader that can handle it.
Determinism is a cost control. Every decision moved from a model to code gets cheaper, faster and repeatable. It also becomes testable, and that makes the next change safe.
Recall first, then make every call earn its place. Build the version that finds everything. Then measure each stage, keep the calls that change the answer and cut the rest, with a test that holds each gain in place.
Say where the answer came from, or do not answer. Aliases, source spans, formula refusals and a value-in-source check all serve one rule. An analyst must be able to put their finger on the page.