An on-prem LLM gateway for a bank that cannot send data out.
On-premise LLM platform for financial institutions, architected in C#/.NET. Multi-provider model routing, RAG pipelines with vector search, multi-agent orchestration — and a hard requirement of zero external data egress. Employer and client names withheld; specifics under NDA.
Regulation set the architecture before I wrote a line.
Banks want LLM capability — document understanding, retrieval-augmented answers, agent workflows — but regulation and risk mean customer data cannot leave the building. Public-cloud AI APIs are a non-starter; every byte must stay on infrastructure the institution controls. That turned the project into an infrastructure problem first and a model problem second.
The gateway's job: give internal teams one API surface for AI capability, run entirely on-premise, with the same reliability bar as any other core-banking dependency.
One gateway in front of local inference.
A .NET gateway owns authentication, request shaping, quotas, and audit logging. Behind it, a model router normalises local providers behind one contract; a RAG pipeline chunks, embeds, and searches internal documentation; an orchestrator splits multi-step questions across bounded, logged agents. Sanitized diagram.
Provider abstraction first
Models change monthly. The routing layer normalises providers behind one contract so swapping engines is configuration, not code.
RAG as a pipeline, not a library call
Chunking, embedding, vector search, and prompt assembly are explicit stages — each one tunable and auditable on its own.
Zero egress as an architectural property
Not a policy document: the deployment has no route to external AI endpoints. Compliance can verify it by reading the network, not the handbook.
C#/.NET, because the institution can operate it
Multi-agent orchestration for workflows where one model call isn't enough — deliberate, bounded, logged — in the stack the bank's own teams can extend and audit.
Throughput and deployment specifics are the client's to share, not mine. The personal-lab equivalents of every moving part here — RAG bot, embeddings store, agent deliberation — run on my own hardware and are documented on the fintech-ai page.