Infrastructure · Finance
On-premise LLM infrastructure for financial services
In financial institutions the constraint is not model capability but where the data goes. Zero-Shot builds inference layers that keep sensitive data in-house, enforce routing as a rule and log every call for audit, with cost reported per unit.
What we build
- 01Routing by data class: sensitive requests to local models, the rest to the cloud
- 02Open-weight models running and scaling on internal hardware
- 03An audit record for every call: who, with which data class, to which model
- 04Quotas, rate limits and cost reporting per team and product
- 05Rehearsed outage and provider-migration scenarios
How it runs
01
Classify the data
Which data class may reach which model is written down. That table is the foundation; the engineering starts after it.
02
Gateway and logging
All calls pass through one door and are logged, in a form the audit team can read independently.
03
Stand up local models
A local model is deployed for the sensitive class and compared against the cloud model on the same evaluation set.
04
Rehearse resilience
Provider outages and hardware failures are exercised, showing the fallback path actually works.
What we measure
- Routing rule violations (target: zero)
- Audit log completeness — calls recorded without gaps
- Gap between local and cloud models on the same evaluation set
- Cost per request and distribution across teams
- p95 latency and time survived during an outage
When this is the wrong fit
This is infrastructure work; we do not build systems that make credit, risk or customer-facing decisions. At low call volume the layer also adds complexity without a return.
Common questions
Does data really stay inside?
For the sensitive class, yes: those requests go to a model on internal hardware and never reach the external network. The behaviour is defined in the routing rules and violations are measured. Which class goes where is approved in writing before setup.
What is recorded for audit?
For every call: the requesting system or person, the data class, the target model, latency and cost. Whether request content is retained follows your own policy; when it is not, the metadata record still exists.
Are open-weight models good enough?
It depends on the task. For summarisation, classification and internal search they usually are; on complex reasoning the gap widens. We report that gap as a measurement on your evaluation set rather than as an assertion.
Other work in this sector
Other services
What we've written on this
- Why AI agent projects failAgents that impress in a demo stall in production. Four observations on tool design, error handling, handing off to a human, and the cost ceiling.
- Is it safe to send company data to AIWhat happens when company data reaches a language model: provider terms, training use, data protection law, cross-border transfer and self-hosting.
- Making LLM costs predictableWhy AI spend surprises you at month end, and how to control it: model tiering, prompt caching, output length, and per-request ceilings.
Last updated: 2026-09-05
Let's talk about this one
Tell us what you want to build and within two days we'll tell you who takes it and how we'd measure it.