An on-prem LLM gateway for a bank that cannot send data out.
On-premise LLM platform for a bank, architected in C#/.NET. Multi-provider model routing, RAG pipelines with vector search, multi-agent orchestration — and a hard requirement of zero external data egress. A freelance engagement: the bank's name stays private and specifics are under NDA.
Regulation set the architecture before I wrote a line.
Banks want LLM capability — document understanding, retrieval-augmented answers, agent workflows — but regulation and risk mean customer data cannot leave the building. Public-cloud AI APIs are a non-starter; every byte must stay on infrastructure the institution controls. That turned the project into an infrastructure problem first and a model problem second.
The gateway's job: give internal teams one API surface for AI capability, run entirely on-premise, with the same reliability bar as any other core-banking dependency.
One gateway in front of local inference.
A .NET gateway owns authentication, request shaping, quotas, and audit logging. Behind it, a model router normalises local providers behind one contract; a RAG pipeline chunks, embeds, and searches internal documentation; an orchestrator splits multi-step questions across bounded, logged agents. Sanitized diagram.
Decisions
Provider abstraction first
Models change monthly. The routing layer normalises providers behind one contract so swapping engines is configuration, not code.
RAG as a pipeline, not a library call
Chunking, embedding, vector search, and prompt assembly are explicit stages — each one tunable and auditable on its own.
Zero egress as an architectural property
Not a policy document: the deployment has no route to external AI endpoints. Compliance can verify it by reading the network, not the handbook.
C#/.NET, because the institution can operate it
The bank's own teams work in .NET. Building the gateway in the same stack means they can extend it, audit it and support it without taking on a new runtime — the orchestration and RAG stages included.
Properties
Throughput and deployment specifics are the client's to share, not mine. The personal-lab equivalents of every moving part here — RAG bot, embeddings store, agent deliberation — run on my own hardware and are documented on the fintech-ai page.